[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:58:15 dist-test-slave-nhpw krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:58:15 dist-test-slave-nhpw krb5kdc[399](info): set up 2 sockets
Sep 26 03:58:15 dist-test-slave-nhpw krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260926 03:58:16.505281 392 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/krb5kdc/test-user.keytab.
Sep 26 03:58:16 dist-test-slave-nhpw krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:58:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:58:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 03:58:16.624271 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33677
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33677
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:58:16.706238 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:58:16.706429 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:58:16.706450 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:58:16.707522 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:58:16.707556 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:58:16.707569 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:58:16.707582 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:58:16.707593 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:58:16.709091 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33677
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:58:16.709285 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:58:16.709463 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:58:16.714577 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:16.714607 412 server_base.cc:1034] running on GCE node
W20260926 03:58:16.714577 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:16.714645 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:16.715055 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:58:16.715348 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:58:16.716492 412 hybrid_clock.cc:648] HybridClock initialized: now 1790395096716474 us; error 37 us; skew 500 ppm
Sep 26 03:58:16 dist-test-slave-nhpw krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395096, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:58:16.721977 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 03:58:16.722405 412 webserver.cc:460] Webserver started at http://127.0.98.62:43811/ using document root <none> and password file <none>
I20260926 03:58:16.722604 412 fs_manager.cc:359] Metadata directory not provided
I20260926 03:58:16.722644 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:58:16.722733 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:58:16.723739 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/minicluster-data/master-0/data/instance:
uuid: "d07b2a2b0bf440dca24cbc299616dded"
format_stamp: "Formatted at 2026-09-26 03:58:16 on dist-test-slave-nhpw"
server_key: "65d5a5b9480aea396e167e702a780ad3"
server_key_iv: "17ccef261ce459149a12472568176369"
server_key_version: "encryptionkey@0"
I20260926 03:58:16.724054 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/minicluster-data/master-0/wal/instance:
uuid: "d07b2a2b0bf440dca24cbc299616dded"
format_stamp: "Formatted at 2026-09-26 03:58:16 on dist-test-slave-nhpw"
server_key: "65d5a5b9480aea396e167e702a780ad3"
server_key_iv: "17ccef261ce459149a12472568176369"
server_key_version: "encryptionkey@0"
I20260926 03:58:16.725451 412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 03:58:16.726179 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:16.726353 412 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 03:58:16.726459 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/minicluster-data/master-0/wal
uuid: "d07b2a2b0bf440dca24cbc299616dded"
format_stamp: "Formatted at 2026-09-26 03:58:16 on dist-test-slave-nhpw"
server_key: "65d5a5b9480aea396e167e702a780ad3"
server_key_iv: "17ccef261ce459149a12472568176369"
server_key_version: "encryptionkey@0"
I20260926 03:58:16.726536 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:58:16.736090 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:58:16.738073 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:58:16.738248 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:58:16.742528 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33677
I20260926 03:58:16.742559 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33677 every 8 connection(s)
I20260926 03:58:16.742925 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
I20260926 03:58:16.743880 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:58:16.746347 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d07b2a2b0bf440dca24cbc299616dded: Bootstrap starting.
I20260926 03:58:16.746954 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d07b2a2b0bf440dca24cbc299616dded: Neither blocks nor log segments found. Creating new log.
I20260926 03:58:16.747246 480 log.cc:824] T 00000000000000000000000000000000 P d07b2a2b0bf440dca24cbc299616dded: Log is configured to *not* fsync() on all Append() calls
I20260926 03:58:16.748075 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d07b2a2b0bf440dca24cbc299616dded: No bootstrap required, opened a new log
I20260926 03:58:16.749264 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 412
I20260926 03:58:16.749372 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/minicluster-data/master-0/wal/instance
I20260926 03:58:16.750125 392 external_mini_cluster.cc:1428] Setting key 4fff8f936220c013443c545a005220f9
I20260926 03:58:16.750218 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P d07b2a2b0bf440dca24cbc299616dded [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d07b2a2b0bf440dca24cbc299616dded" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33677 } }
I20260926 03:58:16.750393 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P d07b2a2b0bf440dca24cbc299616dded [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:58:16.750429 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P d07b2a2b0bf440dca24cbc299616dded [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d07b2a2b0bf440dca24cbc299616dded, State: Initialized, Role: FOLLOWER
I20260926 03:58:16.750576 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P d07b2a2b0bf440dca24cbc299616dded [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d07b2a2b0bf440dca24cbc299616dded" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33677 } }
I20260926 03:58:16.750651 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P d07b2a2b0bf440dca24cbc299616dded [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:58:16.750687 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P d07b2a2b0bf440dca24cbc299616dded [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:58:16.750754 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d07b2a2b0bf440dca24cbc299616dded [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:58:16.751843 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P d07b2a2b0bf440dca24cbc299616dded [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d07b2a2b0bf440dca24cbc299616dded" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33677 } }
I20260926 03:58:16.751961 480 leader_election.cc:302] T 00000000000000000000000000000000 P d07b2a2b0bf440dca24cbc299616dded [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d07b2a2b0bf440dca24cbc299616dded; no voters:
I20260926 03:58:16.752111 480 leader_election.cc:288] T 00000000000000000000000000000000 P d07b2a2b0bf440dca24cbc299616dded [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:58:16.752133 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d07b2a2b0bf440dca24cbc299616dded [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:58:16.752377 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P d07b2a2b0bf440dca24cbc299616dded [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:58:16.752359 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P d07b2a2b0bf440dca24cbc299616dded [term 1 LEADER]: Becoming Leader. State: Replica: d07b2a2b0bf440dca24cbc299616dded, State: Running, Role: LEADER
I20260926 03:58:16.752749 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P d07b2a2b0bf440dca24cbc299616dded [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d07b2a2b0bf440dca24cbc299616dded" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33677 } }
I20260926 03:58:16.753194 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P d07b2a2b0bf440dca24cbc299616dded [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d07b2a2b0bf440dca24cbc299616dded" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d07b2a2b0bf440dca24cbc299616dded" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33677 } } }
I20260926 03:58:16.753201 487 sys_catalog.cc:455] T 00000000000000000000000000000000 P d07b2a2b0bf440dca24cbc299616dded [sys.catalog]: SysCatalogTable state changed. Reason: New leader d07b2a2b0bf440dca24cbc299616dded. Latest consensus state: current_term: 1 leader_uuid: "d07b2a2b0bf440dca24cbc299616dded" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d07b2a2b0bf440dca24cbc299616dded" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33677 } } }
I20260926 03:58:16.753300 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P d07b2a2b0bf440dca24cbc299616dded [sys.catalog]: This master's current role is: LEADER
I20260926 03:58:16.753317 487 sys_catalog.cc:458] T 00000000000000000000000000000000 P d07b2a2b0bf440dca24cbc299616dded [sys.catalog]: This master's current role is: LEADER
I20260926 03:58:16.753515 494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:58:16.754088 494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:58:16.755587 494 catalog_manager.cc:1347] Generated new cluster ID: 258d1929fbd94521a4b0b2ad22984d42
I20260926 03:58:16.755642 494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 03:58:16 dist-test-slave-nhpw krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 03:58:16.764940 494 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:58:16.765631 494 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:58:16.766009 484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:16.751103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53302 (local address 127.0.98.62:33677)
0926 03:58:16.751465 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:16.751472 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:16.751621 (+ 149us) server_negotiation.cc:407] Connection header received
0926 03:58:16.754241 (+ 2620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:16.754249 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:16.754312 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:16.754434 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:16.756329 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:16.757323 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:16.758347 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:16.758763 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:16.758894 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:16.762282 (+ 3388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:16.762288 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:16.762289 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:16.762303 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:16.763986 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:16.764438 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:16.764441 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:16.764441 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:16.764556 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:16.764911 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:16.764912 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:16.764912 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:16.765181 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:16.765362 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:16.765633 (+ 271us) server_negotiation.cc:299] Negotiation successful
0926 03:58:16.765777 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":146,"threads_started":1}
I20260926 03:58:16.775581 494 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d07b2a2b0bf440dca24cbc299616dded: Generated new TSK 0
I20260926 03:58:16.775794 494 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 03:58:16.807914 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33677
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:58:16.892921 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:58:16.893165 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:58:16.893208 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:58:16.894459 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:58:16.894536 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:58:16.894583 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 03:58:16.896140 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33677
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:58:16.896427 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:58:16.896705 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:58:16.901772 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:16.901798 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:16.901806 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:16.902076 506 server_base.cc:1034] running on GCE node
I20260926 03:58:16.902268 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:58:16.902482 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:58:16.903632 506 hybrid_clock.cc:648] HybridClock initialized: now 1790395096903612 us; error 37 us; skew 500 ppm
Sep 26 03:58:16 dist-test-slave-nhpw krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395096, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:58:16.906291 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 03:58:16.906764 506 webserver.cc:460] Webserver started at http://127.0.98.1:44785/ using document root <none> and password file <none>
I20260926 03:58:16.907004 506 fs_manager.cc:359] Metadata directory not provided
I20260926 03:58:16.907050 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:58:16.907194 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:58:16.908242 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/minicluster-data/ts-0/data/instance:
uuid: "20f1485027d04666a26285af03f3dae3"
format_stamp: "Formatted at 2026-09-26 03:58:16 on dist-test-slave-nhpw"
server_key: "b2f9453a8ae02c56093d575a7ec5f906"
server_key_iv: "6007dfdd5a76f66de71cb12f15518a7f"
server_key_version: "encryptionkey@0"
I20260926 03:58:16.908607 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/minicluster-data/ts-0/wal/instance:
uuid: "20f1485027d04666a26285af03f3dae3"
format_stamp: "Formatted at 2026-09-26 03:58:16 on dist-test-slave-nhpw"
server_key: "b2f9453a8ae02c56093d575a7ec5f906"
server_key_iv: "6007dfdd5a76f66de71cb12f15518a7f"
server_key_version: "encryptionkey@0"
I20260926 03:58:16.910104 506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 03:58:16.910912 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:16.911154 506 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 03:58:16.911262 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/minicluster-data/ts-0/wal
uuid: "20f1485027d04666a26285af03f3dae3"
format_stamp: "Formatted at 2026-09-26 03:58:16 on dist-test-slave-nhpw"
server_key: "b2f9453a8ae02c56093d575a7ec5f906"
server_key_iv: "6007dfdd5a76f66de71cb12f15518a7f"
server_key_version: "encryptionkey@0"
I20260926 03:58:16.911368 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:58:16.932588 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:58:16.934751 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:58:16.934937 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:58:16.935184 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:58:16.935516 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:58:16.935575 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:16.935609 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:58:16.935638 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:16.941802 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34829
I20260926 03:58:16.941854 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34829 every 8 connection(s)
I20260926 03:58:16.942178 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
I20260926 03:58:16.943354 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 506
I20260926 03:58:16.943440 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/minicluster-data/ts-0/wal/instance
I20260926 03:58:16.943598 392 external_mini_cluster.cc:1428] Setting key 98d36f10a0ca067c23177d7054efd32c
Sep 26 03:58:16 dist-test-slave-nhpw krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395096, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 03:58:16.953251 484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:16.942966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58371 (local address 127.0.98.62:33677)
0926 03:58:16.943059 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:16.943062 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:16.943248 (+ 186us) server_negotiation.cc:407] Connection header received
0926 03:58:16.943910 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:16.943912 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:16.943923 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:16.943981 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:16.945065 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:16.945819 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:16.946894 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:16.947240 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:16.947323 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:16.949970 (+ 2647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:16.949972 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:16.949972 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:16.949982 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:16.951620 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:16.952123 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:16.952125 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:16.952125 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:16.952186 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:16.952449 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:16.952450 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:16.952450 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:16.952599 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:16.952722 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:16.953115 (+ 393us) server_negotiation.cc:299] Negotiation successful
0926 03:58:16.953186 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 03:58:16.953528 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:16.942924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33677 (local address 127.0.98.1:58371)
0926 03:58:16.943111 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:16.943162 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:16.943800 (+ 638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:16.944060 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:16.944064 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:16.944720 (+ 656us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:16.944938 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:16.944943 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:16.945932 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:16.945934 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:16.946787 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:16.946789 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:16.947372 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:16.947373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:16.947518 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:16.948131 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:16.948139 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:16.949869 (+ 1730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:58:16.951729 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:16.951735 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:16.951737 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:16.952046 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:16.952280 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:16.952281 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:16.952282 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:16.952386 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:16.952719 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:58:16.952728 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:58:16.952904 (+ 176us) client_negotiation.cc:769] Sending connection context
0926 03:58:16.953097 (+ 193us) client_negotiation.cc:240] Negotiation successful
0926 03:58:16.953246 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":95,"threads_started":1}
I20260926 03:58:16.954155 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33677
I20260926 03:58:16.954278 635 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260926 03:58:16.954491 635 heartbeater.cc:511] Master 127.0.98.62:33677 requested a full tablet report, sending...
I20260926 03:58:16.955044 444 ts_manager.cc:194] Registered new tserver with Master: 20f1485027d04666a26285af03f3dae3 (127.0.98.1:34829)
I20260926 03:58:16.955724 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58371
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 03:58:16.973222 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33677
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:58:17.054062 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:58:17.054315 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:58:17.054359 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:58:17.055485 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:58:17.055526 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:58:17.055553 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 03:58:17.056973 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33677
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:58:17.057161 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:58:17.057340 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:58:17.062408 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:17.062486 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:17.062531 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:17.062903 640 server_base.cc:1034] running on GCE node
I20260926 03:58:17.063146 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:58:17.063390 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:58:17.064554 640 hybrid_clock.cc:648] HybridClock initialized: now 1790395097064550 us; error 41 us; skew 500 ppm
Sep 26 03:58:17 dist-test-slave-nhpw krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395097, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:58:17.067008 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 03:58:17.067463 640 webserver.cc:460] Webserver started at http://127.0.98.2:45523/ using document root <none> and password file <none>
I20260926 03:58:17.067689 640 fs_manager.cc:359] Metadata directory not provided
I20260926 03:58:17.067734 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:58:17.067864 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:58:17.068984 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/minicluster-data/ts-1/data/instance:
uuid: "335664005c61420b887ac996faff50b3"
format_stamp: "Formatted at 2026-09-26 03:58:17 on dist-test-slave-nhpw"
server_key: "f6662119c6755f36653b5e0a65c7f2ce"
server_key_iv: "7a0a939a280c2c6baf46b6f922d52e4e"
server_key_version: "encryptionkey@0"
I20260926 03:58:17.069337 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/minicluster-data/ts-1/wal/instance:
uuid: "335664005c61420b887ac996faff50b3"
format_stamp: "Formatted at 2026-09-26 03:58:17 on dist-test-slave-nhpw"
server_key: "f6662119c6755f36653b5e0a65c7f2ce"
server_key_iv: "7a0a939a280c2c6baf46b6f922d52e4e"
server_key_version: "encryptionkey@0"
I20260926 03:58:17.070652 640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 03:58:17.071369 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:17.071547 640 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 03:58:17.071646 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/minicluster-data/ts-1/wal
uuid: "335664005c61420b887ac996faff50b3"
format_stamp: "Formatted at 2026-09-26 03:58:17 on dist-test-slave-nhpw"
server_key: "f6662119c6755f36653b5e0a65c7f2ce"
server_key_iv: "7a0a939a280c2c6baf46b6f922d52e4e"
server_key_version: "encryptionkey@0"
I20260926 03:58:17.071725 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:58:17.084508 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:58:17.086450 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:58:17.086618 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:58:17.086864 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:58:17.087172 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:58:17.087227 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:17.087272 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:58:17.087293 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:17.093927 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41199
I20260926 03:58:17.094027 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41199 every 8 connection(s)
I20260926 03:58:17.094363 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
I20260926 03:58:17.098140 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 640
I20260926 03:58:17.098258 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/minicluster-data/ts-1/wal/instance
I20260926 03:58:17.098424 392 external_mini_cluster.cc:1428] Setting key dc4c0b33ec5f751c4f1174204fedd8e4
Sep 26 03:58:17 dist-test-slave-nhpw krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395097, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 03:58:17.104974 484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:17.095185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51309 (local address 127.0.98.62:33677)
0926 03:58:17.095258 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:17.095260 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:17.095466 (+ 206us) server_negotiation.cc:407] Connection header received
0926 03:58:17.095961 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:17.095962 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:17.095971 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:17.096029 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:17.096956 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:17.097445 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:17.098835 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:17.099050 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:17.099141 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:17.101784 (+ 2643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:17.101785 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:17.101786 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:17.101793 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:17.103284 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:17.103865 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:17.103867 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:17.103867 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:17.103927 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:17.104256 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:17.104257 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:17.104257 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:17.104378 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:17.104445 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:17.104830 (+ 385us) server_negotiation.cc:299] Negotiation successful
0926 03:58:17.104908 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 03:58:17.105243 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:17.095150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33677 (local address 127.0.98.2:51309)
0926 03:58:17.095335 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:17.095375 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:17.095868 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:17.096089 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:17.096092 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:17.096681 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:17.096878 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:17.096883 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:17.097553 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:17.097554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:17.098716 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:17.098720 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:17.099149 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:17.099150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:17.099284 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:17.099840 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:17.099848 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:17.101679 (+ 1831us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:58:17.103395 (+ 1716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:17.103403 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:17.103405 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:17.103741 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:17.104008 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:17.104009 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:17.104010 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:17.104164 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:17.104460 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:58:17.104467 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:58:17.104668 (+ 201us) client_negotiation.cc:769] Sending connection context
0926 03:58:17.104798 (+ 130us) client_negotiation.cc:240] Negotiation successful
0926 03:58:17.104991 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":117,"threads_started":1}
I20260926 03:58:17.105788 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33677
I20260926 03:58:17.105885 769 heartbeater.cc:464] Registering TS with master...
I20260926 03:58:17.106094 769 heartbeater.cc:511] Master 127.0.98.62:33677 requested a full tablet report, sending...
I20260926 03:58:17.106492 444 ts_manager.cc:194] Registered new tserver with Master: 335664005c61420b887ac996faff50b3 (127.0.98.2:41199)
I20260926 03:58:17.106871 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51309
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 03:58:17.124394 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33677
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:58:17.206984 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:58:17.207250 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:58:17.207289 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:58:17.208472 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:58:17.208549 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:58:17.208595 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 03:58:17.210071 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33677
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:58:17.210345 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:58:17.210583 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:58:17.216121 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:17.216166 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:17.216140 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:17.216169 774 server_base.cc:1034] running on GCE node
I20260926 03:58:17.216521 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:58:17.216781 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:58:17.217944 774 hybrid_clock.cc:648] HybridClock initialized: now 1790395097217927 us; error 32 us; skew 500 ppm
Sep 26 03:58:17 dist-test-slave-nhpw krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395097, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:58:17.220512 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 03:58:17.221048 774 webserver.cc:460] Webserver started at http://127.0.98.3:41255/ using document root <none> and password file <none>
I20260926 03:58:17.221279 774 fs_manager.cc:359] Metadata directory not provided
I20260926 03:58:17.221323 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:58:17.221516 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:58:17.222580 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/minicluster-data/ts-2/data/instance:
uuid: "c418a5e50c73415b8547f18d29a2051d"
format_stamp: "Formatted at 2026-09-26 03:58:17 on dist-test-slave-nhpw"
server_key: "cfd3268a8e0fb07e000a092798a0777c"
server_key_iv: "1da479007ceb6e87d5d8616aef6be89f"
server_key_version: "encryptionkey@0"
I20260926 03:58:17.222944 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c418a5e50c73415b8547f18d29a2051d"
format_stamp: "Formatted at 2026-09-26 03:58:17 on dist-test-slave-nhpw"
server_key: "cfd3268a8e0fb07e000a092798a0777c"
server_key_iv: "1da479007ceb6e87d5d8616aef6be89f"
server_key_version: "encryptionkey@0"
I20260926 03:58:17.224246 774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 03:58:17.225096 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:17.225274 774 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:17.225376 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/minicluster-data/ts-2/wal
uuid: "c418a5e50c73415b8547f18d29a2051d"
format_stamp: "Formatted at 2026-09-26 03:58:17 on dist-test-slave-nhpw"
server_key: "cfd3268a8e0fb07e000a092798a0777c"
server_key_iv: "1da479007ceb6e87d5d8616aef6be89f"
server_key_version: "encryptionkey@0"
I20260926 03:58:17.225452 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:58:17.234233 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:58:17.236567 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:58:17.236756 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:58:17.236996 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:58:17.237310 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:58:17.237365 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:17.237406 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:58:17.237435 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:17.244423 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33621
I20260926 03:58:17.244467 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33621 every 8 connection(s)
I20260926 03:58:17.244879 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
I20260926 03:58:17.248914 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 774
I20260926 03:58:17.249032 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790395095474643-392-0/minicluster-data/ts-2/wal/instance
I20260926 03:58:17.249221 392 external_mini_cluster.cc:1428] Setting key e5f90ca0a4259a542a20230db28a5d56
Sep 26 03:58:17 dist-test-slave-nhpw krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395097, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 03:58:17.255183 484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:17.245605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44789 (local address 127.0.98.62:33677)
0926 03:58:17.245676 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:17.245678 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:17.245885 (+ 207us) server_negotiation.cc:407] Connection header received
0926 03:58:17.246413 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:17.246414 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:17.246428 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:17.246484 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:17.247460 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:17.247987 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:17.249166 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:17.249419 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:17.249507 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:17.251945 (+ 2438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:17.251946 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:17.251946 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:17.251953 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:17.253578 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:17.254151 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:17.254152 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:17.254153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:17.254218 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:17.254471 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:17.254473 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:17.254473 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:17.254596 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:17.254676 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:17.255002 (+ 326us) server_negotiation.cc:299] Negotiation successful
0926 03:58:17.255088 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 03:58:17.255369 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:17.245578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33677 (local address 127.0.98.3:44789)
0926 03:58:17.245736 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:17.245784 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:17.246305 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:17.246560 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:17.246564 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:17.247150 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:17.247350 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:17.247355 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:17.248087 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:17.248088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:17.249011 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:17.249013 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:17.249516 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:17.249517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:17.249839 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:17.250410 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:17.250418 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:17.251845 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:58:17.253679 (+ 1834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:17.253823 (+ 144us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:17.253825 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:17.254085 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:17.254279 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:17.254280 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:17.254280 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:17.254382 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:17.254668 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:58:17.254674 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:58:17.254820 (+ 146us) client_negotiation.cc:769] Sending connection context
0926 03:58:17.254949 (+ 129us) client_negotiation.cc:240] Negotiation successful
0926 03:58:17.255122 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":73,"threads_started":1}
I20260926 03:58:17.255865 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33677
I20260926 03:58:17.255972 903 heartbeater.cc:464] Registering TS with master...
I20260926 03:58:17.256150 903 heartbeater.cc:511] Master 127.0.98.62:33677 requested a full tablet report, sending...
I20260926 03:58:17.256520 444 ts_manager.cc:194] Registered new tserver with Master: c418a5e50c73415b8547f18d29a2051d (127.0.98.3:33621)
I20260926 03:58:17.256916 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44789
I20260926 03:58:17.261334 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 03:58:17 dist-test-slave-nhpw krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395097, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 03:58:17 dist-test-slave-nhpw krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395097, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 03:58:17.285625 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:17.277655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48972 (local address 127.0.98.1:34829)
0926 03:58:17.277845 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:17.277849 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:17.277861 (+ 12us) server_negotiation.cc:407] Connection header received
0926 03:58:17.277886 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:17.277888 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:17.277902 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:17.277968 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:17.278509 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:17.279246 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:17.279979 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:17.280282 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:17.280336 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:17.282676 (+ 2340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:17.282692 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:17.282692 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:17.282700 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:17.284310 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:17.284740 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:17.284745 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:17.284745 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:17.284826 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:17.285045 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:17.285046 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:17.285046 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:17.285190 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:17.285371 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:17.285415 (+ 44us) server_negotiation.cc:299] Negotiation successful
0926 03:58:17.285495 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":85,"threads_started":1}
W20260926 03:58:17.286010 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48972
Sep 26 03:58:17 dist-test-slave-nhpw krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395097, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 03:58:17 dist-test-slave-nhpw krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395097, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 03:58:17.308732 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:17.301607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48986 (local address 127.0.98.1:34829)
0926 03:58:17.301693 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:17.301695 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:17.301721 (+ 26us) server_negotiation.cc:407] Connection header received
0926 03:58:17.301803 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:17.301804 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:17.301817 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:17.301872 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:17.302428 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:17.302918 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:17.303635 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:17.303849 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:17.303905 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:17.306224 (+ 2319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:17.306226 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:17.306226 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:17.306235 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:17.307601 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:17.307961 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:17.307962 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:17.307962 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:17.308015 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:17.308175 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:17.308175 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:17.308175 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:17.308263 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:17.308355 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:17.308489 (+ 134us) server_negotiation.cc:299] Negotiation successful
0926 03:58:17.308556 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 03:58:17.309374 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 506
I20260926 03:58:17.314242 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 640
I20260926 03:58:17.320125 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 774
I20260926 03:58:17.324159 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 412
2026-09-26T03:58:17Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1856 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:58:17 dist-test-slave-nhpw krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:58:17 dist-test-slave-nhpw krb5kdc[917](info): set up 2 sockets
Sep 26 03:58:17 dist-test-slave-nhpw krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260926 03:58:18.360076 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/krb5kdc/test-user.keytab.
Sep 26 03:58:18 dist-test-slave-nhpw krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:58:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:58:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 03:58:18.471674 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37109
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37109
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:58:18.550740 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:58:18.550980 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:58:18.551017 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:58:18.552155 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:58:18.552222 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:58:18.552248 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:58:18.552278 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:58:18.552302 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:58:18.553815 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37109
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:58:18.554091 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:58:18.554312 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:58:18.559068 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:18.559072 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:18.559037 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:18.559079 930 server_base.cc:1034] running on GCE node
I20260926 03:58:18.559420 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:58:18.559733 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:58:18.560889 930 hybrid_clock.cc:648] HybridClock initialized: now 1790395098560862 us; error 39 us; skew 500 ppm
Sep 26 03:58:18 dist-test-slave-nhpw krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395098, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:58:18.563297 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 03:58:18.563695 930 webserver.cc:460] Webserver started at http://127.0.98.62:40953/ using document root <none> and password file <none>
I20260926 03:58:18.563872 930 fs_manager.cc:359] Metadata directory not provided
I20260926 03:58:18.563915 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:58:18.563999 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:58:18.565042 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/minicluster-data/master-0/data/instance:
uuid: "d64c27006f6f46dfa88a98b1f81ea07a"
format_stamp: "Formatted at 2026-09-26 03:58:18 on dist-test-slave-nhpw"
server_key: "1e4f2aae0bba3c5f7cec7e9bb1f0f832"
server_key_iv: "f34fdbeed18971c92b721187460cfdf9"
server_key_version: "encryptionkey@0"
I20260926 03:58:18.565323 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/minicluster-data/master-0/wal/instance:
uuid: "d64c27006f6f46dfa88a98b1f81ea07a"
format_stamp: "Formatted at 2026-09-26 03:58:18 on dist-test-slave-nhpw"
server_key: "1e4f2aae0bba3c5f7cec7e9bb1f0f832"
server_key_iv: "f34fdbeed18971c92b721187460cfdf9"
server_key_version: "encryptionkey@0"
I20260926 03:58:18.566550 930 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 03:58:18.567181 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:18.567351 930 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 03:58:18.567421 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/minicluster-data/master-0/wal
uuid: "d64c27006f6f46dfa88a98b1f81ea07a"
format_stamp: "Formatted at 2026-09-26 03:58:18 on dist-test-slave-nhpw"
server_key: "1e4f2aae0bba3c5f7cec7e9bb1f0f832"
server_key_iv: "f34fdbeed18971c92b721187460cfdf9"
server_key_version: "encryptionkey@0"
I20260926 03:58:18.567471 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:58:18.584724 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:58:18.587020 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:58:18.587132 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:58:18.591228 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37109
I20260926 03:58:18.591631 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
I20260926 03:58:18.591642 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37109 every 8 connection(s)
I20260926 03:58:18.592345 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:58:18.594758 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d64c27006f6f46dfa88a98b1f81ea07a: Bootstrap starting.
I20260926 03:58:18.595319 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d64c27006f6f46dfa88a98b1f81ea07a: Neither blocks nor log segments found. Creating new log.
I20260926 03:58:18.595578 998 log.cc:824] T 00000000000000000000000000000000 P d64c27006f6f46dfa88a98b1f81ea07a: Log is configured to *not* fsync() on all Append() calls
I20260926 03:58:18.596175 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 930
I20260926 03:58:18.596275 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/minicluster-data/master-0/wal/instance
I20260926 03:58:18.596376 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d64c27006f6f46dfa88a98b1f81ea07a: No bootstrap required, opened a new log
I20260926 03:58:18.596428 392 external_mini_cluster.cc:1428] Setting key 346500842190167556c654b19bdad218
I20260926 03:58:18.598857 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P d64c27006f6f46dfa88a98b1f81ea07a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d64c27006f6f46dfa88a98b1f81ea07a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37109 } }
I20260926 03:58:18.598994 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P d64c27006f6f46dfa88a98b1f81ea07a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:58:18.599025 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P d64c27006f6f46dfa88a98b1f81ea07a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d64c27006f6f46dfa88a98b1f81ea07a, State: Initialized, Role: FOLLOWER
I20260926 03:58:18.599155 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P d64c27006f6f46dfa88a98b1f81ea07a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d64c27006f6f46dfa88a98b1f81ea07a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37109 } }
I20260926 03:58:18.599226 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P d64c27006f6f46dfa88a98b1f81ea07a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:58:18.599258 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P d64c27006f6f46dfa88a98b1f81ea07a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:58:18.599309 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d64c27006f6f46dfa88a98b1f81ea07a [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:58:18.599888 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P d64c27006f6f46dfa88a98b1f81ea07a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d64c27006f6f46dfa88a98b1f81ea07a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37109 } }
I20260926 03:58:18.599974 998 leader_election.cc:302] T 00000000000000000000000000000000 P d64c27006f6f46dfa88a98b1f81ea07a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d64c27006f6f46dfa88a98b1f81ea07a; no voters:
I20260926 03:58:18.600165 998 leader_election.cc:288] T 00000000000000000000000000000000 P d64c27006f6f46dfa88a98b1f81ea07a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:58:18.600190 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d64c27006f6f46dfa88a98b1f81ea07a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:58:18.600312 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P d64c27006f6f46dfa88a98b1f81ea07a [term 1 LEADER]: Becoming Leader. State: Replica: d64c27006f6f46dfa88a98b1f81ea07a, State: Running, Role: LEADER
I20260926 03:58:18.600395 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P d64c27006f6f46dfa88a98b1f81ea07a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d64c27006f6f46dfa88a98b1f81ea07a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37109 } }
I20260926 03:58:18.600425 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P d64c27006f6f46dfa88a98b1f81ea07a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:58:18.600715 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P d64c27006f6f46dfa88a98b1f81ea07a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d64c27006f6f46dfa88a98b1f81ea07a. Latest consensus state: current_term: 1 leader_uuid: "d64c27006f6f46dfa88a98b1f81ea07a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d64c27006f6f46dfa88a98b1f81ea07a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37109 } } }
I20260926 03:58:18.600797 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P d64c27006f6f46dfa88a98b1f81ea07a [sys.catalog]: This master's current role is: LEADER
I20260926 03:58:18.600730 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P d64c27006f6f46dfa88a98b1f81ea07a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d64c27006f6f46dfa88a98b1f81ea07a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d64c27006f6f46dfa88a98b1f81ea07a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37109 } } }
I20260926 03:58:18.600965 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P d64c27006f6f46dfa88a98b1f81ea07a [sys.catalog]: This master's current role is: LEADER
I20260926 03:58:18.601015 1008 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:58:18.601555 1008 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:58:18.603291 1008 catalog_manager.cc:1347] Generated new cluster ID: 093b518a642e4d399b5d23bf44df58f6
I20260926 03:58:18.603359 1008 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 03:58:18 dist-test-slave-nhpw krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 03:58:18.608814 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:18.596999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54762 (local address 127.0.98.62:37109)
0926 03:58:18.597183 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:18.597191 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:18.597223 (+ 32us) server_negotiation.cc:407] Connection header received
0926 03:58:18.597768 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:18.597775 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:18.597822 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:18.597937 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:18.598742 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:18.600188 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:18.601354 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:18.602057 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:18.602477 (+ 420us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:18.604935 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:18.604939 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:18.604939 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:18.604952 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:18.606958 (+ 2006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:18.607360 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:18.607364 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:18.607364 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:18.607470 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:18.607671 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:18.607672 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:18.607673 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:18.607989 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:18.608156 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:18.608261 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 03:58:18.608526 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":88,"threads_started":1}
I20260926 03:58:18.609329 1008 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:58:18.609820 1008 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:58:18.612802 1008 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d64c27006f6f46dfa88a98b1f81ea07a: Generated new TSK 0
I20260926 03:58:18.612969 1008 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 03:58:18.643605 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37109
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:58:18.726451 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:58:18.726668 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:58:18.726701 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:58:18.727962 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:58:18.728001 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:58:18.728029 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 03:58:18.729578 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37109
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:58:18.729833 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:58:18.730087 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:58:18.735203 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:18.735200 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:18.735219 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:18.735455 1024 server_base.cc:1034] running on GCE node
I20260926 03:58:18.735718 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:58:18.735972 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:58:18.737120 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790395098737111 us; error 33 us; skew 500 ppm
Sep 26 03:58:18 dist-test-slave-nhpw krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395098, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:58:18.739861 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 03:58:18.740312 1024 webserver.cc:460] Webserver started at http://127.0.98.1:40279/ using document root <none> and password file <none>
I20260926 03:58:18.740548 1024 fs_manager.cc:359] Metadata directory not provided
I20260926 03:58:18.740597 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:58:18.740779 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:58:18.741832 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/minicluster-data/ts-0/data/instance:
uuid: "c9658db1b2f444d3bf5be229c75e279c"
format_stamp: "Formatted at 2026-09-26 03:58:18 on dist-test-slave-nhpw"
server_key: "35c3e7a0c163c3121b3d38cfcf343a62"
server_key_iv: "92f5526bf9b90f8a9ecc89726c798d82"
server_key_version: "encryptionkey@0"
I20260926 03:58:18.742198 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c9658db1b2f444d3bf5be229c75e279c"
format_stamp: "Formatted at 2026-09-26 03:58:18 on dist-test-slave-nhpw"
server_key: "35c3e7a0c163c3121b3d38cfcf343a62"
server_key_iv: "92f5526bf9b90f8a9ecc89726c798d82"
server_key_version: "encryptionkey@0"
I20260926 03:58:18.743527 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 03:58:18.744243 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:18.744418 1024 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 03:58:18.744524 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/minicluster-data/ts-0/wal
uuid: "c9658db1b2f444d3bf5be229c75e279c"
format_stamp: "Formatted at 2026-09-26 03:58:18 on dist-test-slave-nhpw"
server_key: "35c3e7a0c163c3121b3d38cfcf343a62"
server_key_iv: "92f5526bf9b90f8a9ecc89726c798d82"
server_key_version: "encryptionkey@0"
I20260926 03:58:18.744599 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:58:18.757085 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:58:18.758930 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:58:18.759097 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:58:18.759323 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:58:18.759655 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:58:18.759714 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:18.759748 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:58:18.759775 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:18.765681 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41073
I20260926 03:58:18.765754 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41073 every 8 connection(s)
I20260926 03:58:18.766088 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
I20260926 03:58:18.768507 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 1024
I20260926 03:58:18.768610 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/minicluster-data/ts-0/wal/instance
I20260926 03:58:18.768823 392 external_mini_cluster.cc:1428] Setting key 1fe9cd8aeb49e938311712e5e51e1048
Sep 26 03:58:18 dist-test-slave-nhpw krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395098, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 03:58:18.776162 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:18.766921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52095 (local address 127.0.98.62:37109)
0926 03:58:18.767008 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:18.767011 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:18.767193 (+ 182us) server_negotiation.cc:407] Connection header received
0926 03:58:18.767717 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:18.767718 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:18.767729 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:18.767796 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:18.768828 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:18.769367 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:18.770441 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:18.770662 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:18.770751 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:18.773183 (+ 2432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:18.773184 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:18.773185 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:18.773192 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:18.774733 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:18.775164 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:18.775165 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:18.775166 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:18.775228 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:18.775491 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:18.775492 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:18.775492 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:18.775612 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:18.775697 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:18.775982 (+ 285us) server_negotiation.cc:299] Negotiation successful
0926 03:58:18.776073 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 03:58:18.776381 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:18.766881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37109 (local address 127.0.98.1:52095)
0926 03:58:18.767060 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:18.767107 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:18.767608 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:18.767882 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:18.767886 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:18.768467 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:18.768742 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:18.768748 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:18.769477 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:18.769478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:18.770348 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:18.770350 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:18.770758 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:18.770760 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:18.770891 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:18.771459 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:18.771467 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:18.773072 (+ 1605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:58:18.774839 (+ 1767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:18.774846 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:18.774848 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:18.775070 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:18.775294 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:18.775295 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:18.775296 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:18.775411 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:18.775688 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:58:18.775695 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:58:18.775838 (+ 143us) client_negotiation.cc:769] Sending connection context
0926 03:58:18.775943 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 03:58:18.776100 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":82,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 03:58:18.776852 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37109
I20260926 03:58:18.776973 1153 heartbeater.cc:464] Registering TS with master...
I20260926 03:58:18.777151 1153 heartbeater.cc:511] Master 127.0.98.62:37109 requested a full tablet report, sending...
I20260926 03:58:18.777734 962 ts_manager.cc:194] Registered new tserver with Master: c9658db1b2f444d3bf5be229c75e279c (127.0.98.1:41073)
I20260926 03:58:18.778350 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52095
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 03:58:18.796089 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37109
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:58:18.877233 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:58:18.877450 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:58:18.877475 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:58:18.878659 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:58:18.878702 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:58:18.878758 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 03:58:18.880209 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37109
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:58:18.880463 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:58:18.880752 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:58:18.885661 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:18.885675 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:18.886034 1158 server_base.cc:1034] running on GCE node
W20260926 03:58:18.886121 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:18.886420 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:58:18.886668 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:58:18.887838 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790395098887825 us; error 36 us; skew 500 ppm
Sep 26 03:58:18 dist-test-slave-nhpw krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395098, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:58:18.890455 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 03:58:18.890901 1158 webserver.cc:460] Webserver started at http://127.0.98.2:36097/ using document root <none> and password file <none>
I20260926 03:58:18.891151 1158 fs_manager.cc:359] Metadata directory not provided
I20260926 03:58:18.891222 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:58:18.891407 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:58:18.892544 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/minicluster-data/ts-1/data/instance:
uuid: "830e8fb530594818bc321eefe9cb70f7"
format_stamp: "Formatted at 2026-09-26 03:58:18 on dist-test-slave-nhpw"
server_key: "e8d845110ecdcad4e897494b9ab5e198"
server_key_iv: "5d8e58ff0e8fbf96c0d50033ebfb2879"
server_key_version: "encryptionkey@0"
I20260926 03:58:18.892961 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/minicluster-data/ts-1/wal/instance:
uuid: "830e8fb530594818bc321eefe9cb70f7"
format_stamp: "Formatted at 2026-09-26 03:58:18 on dist-test-slave-nhpw"
server_key: "e8d845110ecdcad4e897494b9ab5e198"
server_key_iv: "5d8e58ff0e8fbf96c0d50033ebfb2879"
server_key_version: "encryptionkey@0"
I20260926 03:58:18.894533 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 03:58:18.895388 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:18.895576 1158 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 03:58:18.895686 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/minicluster-data/ts-1/wal
uuid: "830e8fb530594818bc321eefe9cb70f7"
format_stamp: "Formatted at 2026-09-26 03:58:18 on dist-test-slave-nhpw"
server_key: "e8d845110ecdcad4e897494b9ab5e198"
server_key_iv: "5d8e58ff0e8fbf96c0d50033ebfb2879"
server_key_version: "encryptionkey@0"
I20260926 03:58:18.895761 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:58:18.913014 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:58:18.915036 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:58:18.915205 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:58:18.915450 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:58:18.915776 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:58:18.915833 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:18.915869 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:58:18.915898 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:18.922196 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34525
I20260926 03:58:18.922240 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34525 every 8 connection(s)
I20260926 03:58:18.922601 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
Sep 26 03:58:18 dist-test-slave-nhpw krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395098, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 03:58:18.930640 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 1158
I20260926 03:58:18.930732 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/minicluster-data/ts-1/wal/instance
I20260926 03:58:18.930904 392 external_mini_cluster.cc:1428] Setting key c2f26f3b24e7e0fec2bd6361b09fcbb2
I20260926 03:58:18.932920 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:18.923352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42875 (local address 127.0.98.62:37109)
0926 03:58:18.923430 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:18.923432 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:18.923578 (+ 146us) server_negotiation.cc:407] Connection header received
0926 03:58:18.924110 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:18.924111 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:18.924121 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:18.924179 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:18.925199 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:18.925676 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:18.926705 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:18.926896 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:18.926947 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:18.929590 (+ 2643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:18.929591 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:18.929592 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:18.929600 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:18.931422 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:18.931902 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:18.931904 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:18.931905 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:18.931968 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:18.932278 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:18.932278 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:18.932279 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:18.932388 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:18.932442 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:18.932789 (+ 347us) server_negotiation.cc:299] Negotiation successful
0926 03:58:18.932856 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 03:58:18.933228 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:18.923302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37109 (local address 127.0.98.2:42875)
0926 03:58:18.923469 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:18.923509 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:18.924009 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:18.924245 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:18.924249 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:18.924924 (+ 675us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:18.925114 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:18.925119 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:18.925791 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:18.925792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:18.926619 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:18.926621 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:18.927498 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:18.927498 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:18.927596 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:18.928135 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:18.928143 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:18.929497 (+ 1354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:58:18.931509 (+ 2012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:18.931515 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:18.931517 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:18.931821 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:18.932050 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:18.932050 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:18.932051 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:18.932195 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:18.932456 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:58:18.932461 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:58:18.932630 (+ 169us) client_negotiation.cc:769] Sending connection context
0926 03:58:18.932773 (+ 143us) client_negotiation.cc:240] Negotiation successful
0926 03:58:18.932952 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":97,"threads_started":1}
I20260926 03:58:18.933692 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37109
I20260926 03:58:18.933813 1287 heartbeater.cc:464] Registering TS with master...
I20260926 03:58:18.934029 1287 heartbeater.cc:511] Master 127.0.98.62:37109 requested a full tablet report, sending...
I20260926 03:58:18.934397 962 ts_manager.cc:194] Registered new tserver with Master: 830e8fb530594818bc321eefe9cb70f7 (127.0.98.2:34525)
I20260926 03:58:18.934921 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42875
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260926 03:58:19.779239 1153 heartbeater.cc:503] Master 127.0.98.62:37109 was elected leader, sending a full tablet report...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab.
W20260926 03:58:19.923070 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.3: real 0.991s user 0.000s sys 0.003s
I20260926 03:58:19.923444 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37109
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:58:19.935801 1287 heartbeater.cc:503] Master 127.0.98.62:37109 was elected leader, sending a full tablet report...
W20260926 03:58:20.010414 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:58:20.010655 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:58:20.010697 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:58:20.012006 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:58:20.012073 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:58:20.012120 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 03:58:20.013690 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37109
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:58:20.013957 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:58:20.014205 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:58:20.021046 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:20.021015 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:20.021065 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:20.021209 1292 server_base.cc:1034] running on GCE node
I20260926 03:58:20.021443 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:58:20.021694 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:58:20.022872 1292 hybrid_clock.cc:648] HybridClock initialized: now 1790395100022860 us; error 30 us; skew 500 ppm
Sep 26 03:58:20 dist-test-slave-nhpw krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395100, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:58:20.026047 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 03:58:20.026520 1292 webserver.cc:460] Webserver started at http://127.0.98.3:39759/ using document root <none> and password file <none>
I20260926 03:58:20.026759 1292 fs_manager.cc:359] Metadata directory not provided
I20260926 03:58:20.026808 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:58:20.026988 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:58:20.028153 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/minicluster-data/ts-2/data/instance:
uuid: "bd686e23a0bb44cf9ffdd713647133b0"
format_stamp: "Formatted at 2026-09-26 03:58:20 on dist-test-slave-nhpw"
server_key: "9d87f3458e75361a1faedaba64e102f9"
server_key_iv: "b7dd509cdc177cf23b82c5901d5c6127"
server_key_version: "encryptionkey@0"
I20260926 03:58:20.028517 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/minicluster-data/ts-2/wal/instance:
uuid: "bd686e23a0bb44cf9ffdd713647133b0"
format_stamp: "Formatted at 2026-09-26 03:58:20 on dist-test-slave-nhpw"
server_key: "9d87f3458e75361a1faedaba64e102f9"
server_key_iv: "b7dd509cdc177cf23b82c5901d5c6127"
server_key_version: "encryptionkey@0"
I20260926 03:58:20.029899 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 03:58:20.030704 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:20.030880 1292 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 03:58:20.030980 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/minicluster-data/ts-2/wal
uuid: "bd686e23a0bb44cf9ffdd713647133b0"
format_stamp: "Formatted at 2026-09-26 03:58:20 on dist-test-slave-nhpw"
server_key: "9d87f3458e75361a1faedaba64e102f9"
server_key_iv: "b7dd509cdc177cf23b82c5901d5c6127"
server_key_version: "encryptionkey@0"
I20260926 03:58:20.031064 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:58:20.045656 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:58:20.048098 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:58:20.048273 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:58:20.048537 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:58:20.048929 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:58:20.048997 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:20.049044 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:58:20.049072 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:20.055056 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37427
I20260926 03:58:20.055104 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37427 every 8 connection(s)
I20260926 03:58:20.055459 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
I20260926 03:58:20.058243 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 1292
I20260926 03:58:20.058350 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790395095474643-392-0/minicluster-data/ts-2/wal/instance
I20260926 03:58:20.058506 392 external_mini_cluster.cc:1428] Setting key b7add96fa45f1c303584f0904ecb28d3
Sep 26 03:58:20 dist-test-slave-nhpw krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395100, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 03:58:20.065692 1424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:20.056431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39405 (local address 127.0.98.62:37109)
0926 03:58:20.056547 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:20.056549 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:20.056561 (+ 12us) server_negotiation.cc:407] Connection header received
0926 03:58:20.057089 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:20.057090 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:20.057104 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:20.057183 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:20.058145 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:20.058964 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:20.060124 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:20.060323 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:20.060386 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:20.062728 (+ 2342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:20.062730 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:20.062731 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:20.062740 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:20.064139 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:20.064565 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:20.064567 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:20.064567 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:20.064695 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:20.064958 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:20.064959 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:20.064959 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:20.065097 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:20.065181 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:20.065478 (+ 297us) server_negotiation.cc:299] Negotiation successful
0926 03:58:20.065572 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":73,"threads_started":1}
I20260926 03:58:20.065881 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:20.056216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37109 (local address 127.0.98.3:39405)
0926 03:58:20.056391 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:20.056434 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:20.056983 (+ 549us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:20.057237 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:20.057241 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:20.057815 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:20.058040 (+ 225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:20.058045 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:20.059060 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:20.059061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:20.060030 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:20.060033 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:20.060404 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:20.060405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:20.060507 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:20.061142 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:20.061152 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:20.062582 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:58:20.064238 (+ 1656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:20.064246 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:20.064248 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:20.064482 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:20.064765 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:20.064766 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:20.064766 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:20.064885 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:20.065168 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:58:20.065174 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:58:20.065323 (+ 149us) client_negotiation.cc:769] Sending connection context
0926 03:58:20.065453 (+ 130us) client_negotiation.cc:240] Negotiation successful
0926 03:58:20.065634 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":96,"threads_started":1}
I20260926 03:58:20.066362 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37109
I20260926 03:58:20.066457 1421 heartbeater.cc:464] Registering TS with master...
I20260926 03:58:20.066599 1421 heartbeater.cc:511] Master 127.0.98.62:37109 requested a full tablet report, sending...
I20260926 03:58:20.066985 962 ts_manager.cc:194] Registered new tserver with Master: bd686e23a0bb44cf9ffdd713647133b0 (127.0.98.3:37427)
I20260926 03:58:20.067307 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39405
I20260926 03:58:20.070254 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:58:20.077883 1424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:20.071998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54778 (local address 127.0.98.62:37109)
0926 03:58:20.072058 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:20.072060 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:20.072168 (+ 108us) server_negotiation.cc:407] Connection header received
0926 03:58:20.072203 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:20.072204 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:20.072212 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:20.072261 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:20.072893 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:20.073349 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:20.074077 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:20.074256 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:20.074306 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:20.075469 (+ 1163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:20.075470 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:20.075471 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:20.075478 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:20.076823 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:20.077117 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:20.077117 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:20.077118 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:20.077165 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:20.077422 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:20.077423 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:20.077423 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:20.077551 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:20.077629 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:20.077758 (+ 129us) server_negotiation.cc:299] Negotiation successful
0926 03:58:20.077819 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 03:58:20.080051 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54778:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:58:20.081631 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:58:20.088063 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:20.084124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45952 (local address 127.0.98.1:41073)
0926 03:58:20.084252 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:20.084257 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:20.084330 (+ 73us) server_negotiation.cc:407] Connection header received
0926 03:58:20.084476 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:20.084478 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:20.084492 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:20.084593 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:58:20.084829 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:20.085620 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:20.086945 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:20.087625 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:20.087711 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:20.087776 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:20.087944 (+ 168us) server_negotiation.cc:299] Negotiation successful
0926 03:58:20.087986 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":49,"threads_started":1}
I20260926 03:58:20.088063 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:20.084037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41073 (local address 127.0.0.1:45952)
0926 03:58:20.084231 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:20.084271 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:20.084350 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:20.084618 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:20.084622 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:20.084624 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:58:20.084717 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:20.084719 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:20.085716 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:20.085717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:20.086835 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:20.086838 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:20.087721 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:20.087722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:20.087809 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:20.087823 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 03:58:20.087924 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 03:58:20.087979 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":84,"threads_started":1}
I20260926 03:58:20.088274 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:20.084062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34525 (local address 127.0.0.1:54696)
0926 03:58:20.084460 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:20.084477 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:20.084544 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:20.084833 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:20.084835 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:20.084837 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:58:20.084883 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:20.084885 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:20.086179 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:20.086180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:20.087069 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:20.087071 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:20.088025 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:20.088026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:20.088089 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:20.088093 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 03:58:20.088146 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 03:58:20.088191 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"mutex_wait_us":2,"thread_start_us":90,"threads_started":1}
I20260926 03:58:20.088485 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:20.084062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37427 (local address 127.0.0.1:39046)
0926 03:58:20.084666 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:20.084676 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:20.084726 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:20.084955 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:20.084956 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:20.084957 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:58:20.085001 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:20.085002 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:20.086242 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:20.086243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:20.087304 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:20.087306 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:20.088260 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:20.088261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:20.088317 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:20.088321 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 03:58:20.088375 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 03:58:20.088421 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"mutex_wait_us":12,"thread_start_us":82,"threads_started":1}
I20260926 03:58:20.088587 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:20.084294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54696 (local address 127.0.98.2:34525)
0926 03:58:20.084427 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:20.084431 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:20.084548 (+ 117us) server_negotiation.cc:407] Connection header received
0926 03:58:20.084627 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:20.084629 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:20.084677 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:20.084748 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:58:20.084961 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:20.086077 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:20.087156 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:20.087864 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:20.087937 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:20.087989 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:20.088442 (+ 453us) server_negotiation.cc:299] Negotiation successful
0926 03:58:20.088497 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":74,"threads_started":1}
I20260926 03:58:20.088821 1439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:20.084294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39046 (local address 127.0.98.3:37427)
0926 03:58:20.084472 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:20.084478 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:20.084735 (+ 257us) server_negotiation.cc:407] Connection header received
0926 03:58:20.084790 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:20.084793 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:20.084809 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:20.084878 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:58:20.085076 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:20.086136 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:20.087389 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:20.088121 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:20.088181 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:20.088230 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:20.088694 (+ 464us) server_negotiation.cc:299] Negotiation successful
0926 03:58:20.088740 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":71,"threads_started":1}
I20260926 03:58:20.089387 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 9331eea4406548ad934f40ce75d343b3 (DEFAULT_TABLE table=test-table [id=68b0756bdec342a9bb7b7251eb7f75cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:58:20.089387 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 9331eea4406548ad934f40ce75d343b3 (DEFAULT_TABLE table=test-table [id=68b0756bdec342a9bb7b7251eb7f75cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:58:20.089498 1355 tablet_service.cc:1467] Processing CreateTablet for tablet 9331eea4406548ad934f40ce75d343b3 (DEFAULT_TABLE table=test-table [id=68b0756bdec342a9bb7b7251eb7f75cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:58:20.089756 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9331eea4406548ad934f40ce75d343b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:58:20.089757 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9331eea4406548ad934f40ce75d343b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:58:20.089865 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9331eea4406548ad934f40ce75d343b3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:58:20.092514 1440 tablet_bootstrap.cc:492] T 9331eea4406548ad934f40ce75d343b3 P bd686e23a0bb44cf9ffdd713647133b0: Bootstrap starting.
I20260926 03:58:20.093063 1442 tablet_bootstrap.cc:492] T 9331eea4406548ad934f40ce75d343b3 P c9658db1b2f444d3bf5be229c75e279c: Bootstrap starting.
I20260926 03:58:20.093356 1440 tablet_bootstrap.cc:654] T 9331eea4406548ad934f40ce75d343b3 P bd686e23a0bb44cf9ffdd713647133b0: Neither blocks nor log segments found. Creating new log.
I20260926 03:58:20.093691 1442 tablet_bootstrap.cc:654] T 9331eea4406548ad934f40ce75d343b3 P c9658db1b2f444d3bf5be229c75e279c: Neither blocks nor log segments found. Creating new log.
I20260926 03:58:20.093765 1440 log.cc:824] T 9331eea4406548ad934f40ce75d343b3 P bd686e23a0bb44cf9ffdd713647133b0: Log is configured to *not* fsync() on all Append() calls
I20260926 03:58:20.094063 1442 log.cc:824] T 9331eea4406548ad934f40ce75d343b3 P c9658db1b2f444d3bf5be229c75e279c: Log is configured to *not* fsync() on all Append() calls
I20260926 03:58:20.094383 1441 tablet_bootstrap.cc:492] T 9331eea4406548ad934f40ce75d343b3 P 830e8fb530594818bc321eefe9cb70f7: Bootstrap starting.
I20260926 03:58:20.094534 1440 tablet_bootstrap.cc:492] T 9331eea4406548ad934f40ce75d343b3 P bd686e23a0bb44cf9ffdd713647133b0: No bootstrap required, opened a new log
I20260926 03:58:20.094605 1440 ts_tablet_manager.cc:1397] T 9331eea4406548ad934f40ce75d343b3 P bd686e23a0bb44cf9ffdd713647133b0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 03:58:20.095453 1441 tablet_bootstrap.cc:654] T 9331eea4406548ad934f40ce75d343b3 P 830e8fb530594818bc321eefe9cb70f7: Neither blocks nor log segments found. Creating new log.
I20260926 03:58:20.095763 1441 log.cc:824] T 9331eea4406548ad934f40ce75d343b3 P 830e8fb530594818bc321eefe9cb70f7: Log is configured to *not* fsync() on all Append() calls
I20260926 03:58:20.096318 1442 tablet_bootstrap.cc:492] T 9331eea4406548ad934f40ce75d343b3 P c9658db1b2f444d3bf5be229c75e279c: No bootstrap required, opened a new log
I20260926 03:58:20.096396 1442 ts_tablet_manager.cc:1397] T 9331eea4406548ad934f40ce75d343b3 P c9658db1b2f444d3bf5be229c75e279c: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 03:58:20.096526 1441 tablet_bootstrap.cc:492] T 9331eea4406548ad934f40ce75d343b3 P 830e8fb530594818bc321eefe9cb70f7: No bootstrap required, opened a new log
I20260926 03:58:20.096613 1441 ts_tablet_manager.cc:1397] T 9331eea4406548ad934f40ce75d343b3 P 830e8fb530594818bc321eefe9cb70f7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 03:58:20.097194 1440 raft_consensus.cc:348] T 9331eea4406548ad934f40ce75d343b3 P bd686e23a0bb44cf9ffdd713647133b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9658db1b2f444d3bf5be229c75e279c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41073 } } peers { permanent_uuid: "830e8fb530594818bc321eefe9cb70f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34525 } } peers { permanent_uuid: "bd686e23a0bb44cf9ffdd713647133b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37427 } }
I20260926 03:58:20.097385 1440 raft_consensus.cc:374] T 9331eea4406548ad934f40ce75d343b3 P bd686e23a0bb44cf9ffdd713647133b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:58:20.097434 1440 raft_consensus.cc:729] T 9331eea4406548ad934f40ce75d343b3 P bd686e23a0bb44cf9ffdd713647133b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd686e23a0bb44cf9ffdd713647133b0, State: Initialized, Role: FOLLOWER
I20260926 03:58:20.097587 1440 consensus_queue.cc:260] T 9331eea4406548ad934f40ce75d343b3 P bd686e23a0bb44cf9ffdd713647133b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9658db1b2f444d3bf5be229c75e279c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41073 } } peers { permanent_uuid: "830e8fb530594818bc321eefe9cb70f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34525 } } peers { permanent_uuid: "bd686e23a0bb44cf9ffdd713647133b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37427 } }
I20260926 03:58:20.097918 1440 ts_tablet_manager.cc:1428] T 9331eea4406548ad934f40ce75d343b3 P bd686e23a0bb44cf9ffdd713647133b0: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260926 03:58:20.098100 1421 heartbeater.cc:503] Master 127.0.98.62:37109 was elected leader, sending a full tablet report...
I20260926 03:58:20.098858 1442 raft_consensus.cc:348] T 9331eea4406548ad934f40ce75d343b3 P c9658db1b2f444d3bf5be229c75e279c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9658db1b2f444d3bf5be229c75e279c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41073 } } peers { permanent_uuid: "830e8fb530594818bc321eefe9cb70f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34525 } } peers { permanent_uuid: "bd686e23a0bb44cf9ffdd713647133b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37427 } }
I20260926 03:58:20.099052 1442 raft_consensus.cc:374] T 9331eea4406548ad934f40ce75d343b3 P c9658db1b2f444d3bf5be229c75e279c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:58:20.099102 1442 raft_consensus.cc:729] T 9331eea4406548ad934f40ce75d343b3 P c9658db1b2f444d3bf5be229c75e279c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9658db1b2f444d3bf5be229c75e279c, State: Initialized, Role: FOLLOWER
I20260926 03:58:20.099013 1441 raft_consensus.cc:348] T 9331eea4406548ad934f40ce75d343b3 P 830e8fb530594818bc321eefe9cb70f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9658db1b2f444d3bf5be229c75e279c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41073 } } peers { permanent_uuid: "830e8fb530594818bc321eefe9cb70f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34525 } } peers { permanent_uuid: "bd686e23a0bb44cf9ffdd713647133b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37427 } }
I20260926 03:58:20.099200 1441 raft_consensus.cc:374] T 9331eea4406548ad934f40ce75d343b3 P 830e8fb530594818bc321eefe9cb70f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:58:20.099237 1441 raft_consensus.cc:729] T 9331eea4406548ad934f40ce75d343b3 P 830e8fb530594818bc321eefe9cb70f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 830e8fb530594818bc321eefe9cb70f7, State: Initialized, Role: FOLLOWER
I20260926 03:58:20.099260 1442 consensus_queue.cc:260] T 9331eea4406548ad934f40ce75d343b3 P c9658db1b2f444d3bf5be229c75e279c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9658db1b2f444d3bf5be229c75e279c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41073 } } peers { permanent_uuid: "830e8fb530594818bc321eefe9cb70f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34525 } } peers { permanent_uuid: "bd686e23a0bb44cf9ffdd713647133b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37427 } }
I20260926 03:58:20.099362 1441 consensus_queue.cc:260] T 9331eea4406548ad934f40ce75d343b3 P 830e8fb530594818bc321eefe9cb70f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9658db1b2f444d3bf5be229c75e279c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41073 } } peers { permanent_uuid: "830e8fb530594818bc321eefe9cb70f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34525 } } peers { permanent_uuid: "bd686e23a0bb44cf9ffdd713647133b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37427 } }
I20260926 03:58:20.099570 1442 ts_tablet_manager.cc:1428] T 9331eea4406548ad934f40ce75d343b3 P c9658db1b2f444d3bf5be229c75e279c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 03:58:20.099637 1441 ts_tablet_manager.cc:1428] T 9331eea4406548ad934f40ce75d343b3 P 830e8fb530594818bc321eefe9cb70f7: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
W20260926 03:58:20.174109 1288 tablet.cc:2369] T 9331eea4406548ad934f40ce75d343b3 P 830e8fb530594818bc321eefe9cb70f7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:58:20.267418 1154 tablet.cc:2369] T 9331eea4406548ad934f40ce75d343b3 P c9658db1b2f444d3bf5be229c75e279c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:58:20.306023 1422 tablet.cc:2369] T 9331eea4406548ad934f40ce75d343b3 P bd686e23a0bb44cf9ffdd713647133b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:58:20.319446 1446 raft_consensus.cc:482] T 9331eea4406548ad934f40ce75d343b3 P bd686e23a0bb44cf9ffdd713647133b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:58:20.319559 1446 raft_consensus.cc:504] T 9331eea4406548ad934f40ce75d343b3 P bd686e23a0bb44cf9ffdd713647133b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9658db1b2f444d3bf5be229c75e279c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41073 } } peers { permanent_uuid: "830e8fb530594818bc321eefe9cb70f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34525 } } peers { permanent_uuid: "bd686e23a0bb44cf9ffdd713647133b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37427 } }
I20260926 03:58:20.320014 1446 leader_election.cc:288] T 9331eea4406548ad934f40ce75d343b3 P bd686e23a0bb44cf9ffdd713647133b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9658db1b2f444d3bf5be229c75e279c (127.0.98.1:41073), 830e8fb530594818bc321eefe9cb70f7 (127.0.98.2:34525)
I20260926 03:58:20.322927 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:20.320171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41073 (local address 127.0.98.3:57111)
0926 03:58:20.320298 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:20.320311 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:20.320374 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:20.320537 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:20.320538 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:20.320540 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:58:20.320616 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:20.320618 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:20.321332 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:20.321333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:20.322129 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:20.322131 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:20.322666 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:20.322667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:20.322743 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:20.322746 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 03:58:20.322797 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 03:58:20.322826 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"mutex_wait_us":27}
I20260926 03:58:20.322947 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:20.320278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57111 (local address 127.0.98.1:41073)
0926 03:58:20.320323 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:20.320325 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:20.320359 (+ 34us) server_negotiation.cc:407] Connection header received
0926 03:58:20.320448 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:20.320449 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:20.320464 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:20.320518 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:58:20.320698 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:20.321246 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:20.322238 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:20.322584 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:20.322638 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:20.322646 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:20.322825 (+ 179us) server_negotiation.cc:299] Negotiation successful
0926 03:58:20.322868 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 03:58:20.322981 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:20.320302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60033 (local address 127.0.98.2:34525)
0926 03:58:20.320431 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:20.320433 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:20.320543 (+ 110us) server_negotiation.cc:407] Connection header received
0926 03:58:20.320570 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:20.320571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:20.320582 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:20.320666 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:58:20.320827 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:20.321273 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:20.322232 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:20.322607 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:20.322704 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:20.322713 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:20.322873 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 03:58:20.322908 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260926 03:58:20.323030 1449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:20.320247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34525 (local address 127.0.98.3:60033)
0926 03:58:20.320434 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:20.320444 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:20.320505 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:20.320678 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:20.320680 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:20.320681 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:58:20.320754 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:20.320756 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:20.321387 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:20.321387 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:20.322131 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:20.322133 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:20.322712 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:20.322713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:20.322777 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:20.322797 (+ 20us) client_negotiation.cc:769] Sending connection context
0926 03:58:20.322855 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 03:58:20.322888 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":2,"thread_start_us":111,"threads_started":1}
I20260926 03:58:20.323268 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9331eea4406548ad934f40ce75d343b3" candidate_uuid: "bd686e23a0bb44cf9ffdd713647133b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9658db1b2f444d3bf5be229c75e279c" is_pre_election: true
I20260926 03:58:20.323410 1107 raft_consensus.cc:2454] T 9331eea4406548ad934f40ce75d343b3 P c9658db1b2f444d3bf5be229c75e279c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd686e23a0bb44cf9ffdd713647133b0 in term 0.
I20260926 03:58:20.323637 1311 leader_election.cc:302] T 9331eea4406548ad934f40ce75d343b3 P bd686e23a0bb44cf9ffdd713647133b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bd686e23a0bb44cf9ffdd713647133b0, c9658db1b2f444d3bf5be229c75e279c; no voters:
I20260926 03:58:20.323755 1446 raft_consensus.cc:2781] T 9331eea4406548ad934f40ce75d343b3 P bd686e23a0bb44cf9ffdd713647133b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:58:20.323812 1446 raft_consensus.cc:482] T 9331eea4406548ad934f40ce75d343b3 P bd686e23a0bb44cf9ffdd713647133b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:58:20.323841 1446 raft_consensus.cc:3037] T 9331eea4406548ad934f40ce75d343b3 P bd686e23a0bb44cf9ffdd713647133b0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:58:20.323894 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9331eea4406548ad934f40ce75d343b3" candidate_uuid: "bd686e23a0bb44cf9ffdd713647133b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "830e8fb530594818bc321eefe9cb70f7" is_pre_election: true
I20260926 03:58:20.324074 1241 raft_consensus.cc:2454] T 9331eea4406548ad934f40ce75d343b3 P 830e8fb530594818bc321eefe9cb70f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd686e23a0bb44cf9ffdd713647133b0 in term 0.
I20260926 03:58:20.407574 1446 raft_consensus.cc:504] T 9331eea4406548ad934f40ce75d343b3 P bd686e23a0bb44cf9ffdd713647133b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9658db1b2f444d3bf5be229c75e279c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41073 } } peers { permanent_uuid: "830e8fb530594818bc321eefe9cb70f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34525 } } peers { permanent_uuid: "bd686e23a0bb44cf9ffdd713647133b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37427 } }
I20260926 03:58:20.407923 1446 leader_election.cc:288] T 9331eea4406548ad934f40ce75d343b3 P bd686e23a0bb44cf9ffdd713647133b0 [CANDIDATE]: Term 1 election: Requested vote from peers c9658db1b2f444d3bf5be229c75e279c (127.0.98.1:41073), 830e8fb530594818bc321eefe9cb70f7 (127.0.98.2:34525)
I20260926 03:58:20.408039 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9331eea4406548ad934f40ce75d343b3" candidate_uuid: "bd686e23a0bb44cf9ffdd713647133b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9658db1b2f444d3bf5be229c75e279c"
I20260926 03:58:20.408051 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9331eea4406548ad934f40ce75d343b3" candidate_uuid: "bd686e23a0bb44cf9ffdd713647133b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "830e8fb530594818bc321eefe9cb70f7"
I20260926 03:58:20.408138 1107 raft_consensus.cc:3037] T 9331eea4406548ad934f40ce75d343b3 P c9658db1b2f444d3bf5be229c75e279c [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:58:20.408175 1241 raft_consensus.cc:3037] T 9331eea4406548ad934f40ce75d343b3 P 830e8fb530594818bc321eefe9cb70f7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:58:20.409056 1107 raft_consensus.cc:2454] T 9331eea4406548ad934f40ce75d343b3 P c9658db1b2f444d3bf5be229c75e279c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd686e23a0bb44cf9ffdd713647133b0 in term 1.
I20260926 03:58:20.409066 1241 raft_consensus.cc:2454] T 9331eea4406548ad934f40ce75d343b3 P 830e8fb530594818bc321eefe9cb70f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd686e23a0bb44cf9ffdd713647133b0 in term 1.
I20260926 03:58:20.409288 1308 leader_election.cc:302] T 9331eea4406548ad934f40ce75d343b3 P bd686e23a0bb44cf9ffdd713647133b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 830e8fb530594818bc321eefe9cb70f7, bd686e23a0bb44cf9ffdd713647133b0; no voters:
I20260926 03:58:20.409430 1446 raft_consensus.cc:2781] T 9331eea4406548ad934f40ce75d343b3 P bd686e23a0bb44cf9ffdd713647133b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:58:20.409624 1446 raft_consensus.cc:686] T 9331eea4406548ad934f40ce75d343b3 P bd686e23a0bb44cf9ffdd713647133b0 [term 1 LEADER]: Becoming Leader. State: Replica: bd686e23a0bb44cf9ffdd713647133b0, State: Running, Role: LEADER
I20260926 03:58:20.409729 1446 consensus_queue.cc:237] T 9331eea4406548ad934f40ce75d343b3 P bd686e23a0bb44cf9ffdd713647133b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9658db1b2f444d3bf5be229c75e279c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41073 } } peers { permanent_uuid: "830e8fb530594818bc321eefe9cb70f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34525 } } peers { permanent_uuid: "bd686e23a0bb44cf9ffdd713647133b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37427 } }
I20260926 03:58:20.410712 961 catalog_manager.cc:5495] T 9331eea4406548ad934f40ce75d343b3 P bd686e23a0bb44cf9ffdd713647133b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to bd686e23a0bb44cf9ffdd713647133b0 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "bd686e23a0bb44cf9ffdd713647133b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9658db1b2f444d3bf5be229c75e279c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "830e8fb530594818bc321eefe9cb70f7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34525 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bd686e23a0bb44cf9ffdd713647133b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37427 } health_report { overall_health: HEALTHY } } }
Sep 26 03:58:20 dist-test-slave-nhpw krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395098, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 03:58:20.443794 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:20.436247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45966 (local address 127.0.98.1:41073)
0926 03:58:20.436338 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:20.436340 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:20.436351 (+ 11us) server_negotiation.cc:407] Connection header received
0926 03:58:20.436416 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:20.436417 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:20.436426 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:20.436483 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:20.437142 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:20.437621 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:20.438359 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:20.438536 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:20.438588 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:20.441024 (+ 2436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:20.441034 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:20.441034 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:20.441041 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:20.442790 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:20.443118 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:20.443121 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:20.443122 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:20.443175 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:20.443336 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:20.443337 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:20.443337 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:20.443466 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:20.443568 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:20.443625 (+ 57us) server_negotiation.cc:299] Negotiation successful
0926 03:58:20.443685 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
Sep 26 03:58:20 dist-test-slave-nhpw krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395100, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 03:58:20 dist-test-slave-nhpw krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395100, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 03:58:20.519711 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:20.512292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45976 (local address 127.0.98.1:41073)
0926 03:58:20.512353 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:20.512355 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:20.512441 (+ 86us) server_negotiation.cc:407] Connection header received
0926 03:58:20.512560 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:20.512561 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:20.512570 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:20.512636 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:20.513265 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:20.513768 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:20.514570 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:20.514762 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:20.514847 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:20.517199 (+ 2352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:20.517201 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:20.517201 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:20.517209 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:20.518497 (+ 1288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:20.518862 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:20.518863 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:20.518864 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:20.518921 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:20.519130 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:20.519131 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:20.519131 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:20.519230 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:20.519342 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:20.519524 (+ 182us) server_negotiation.cc:299] Negotiation successful
0926 03:58:20.519601 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
Sep 26 03:58:20 dist-test-slave-nhpw krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 03:58:20 dist-test-slave-nhpw krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 03:58:20.618587 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:20.610679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45990 (local address 127.0.98.1:41073)
0926 03:58:20.610793 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:20.610796 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:20.610871 (+ 75us) server_negotiation.cc:407] Connection header received
0926 03:58:20.610935 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:20.610936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:20.610948 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:20.611004 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:20.611604 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:20.612391 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:20.613308 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:20.613633 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:20.613718 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:20.616347 (+ 2629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:20.616349 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:20.616349 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:20.616355 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:20.617603 (+ 1248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:20.617946 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:20.617948 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:20.617948 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:20.618004 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:20.618189 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:20.618190 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:20.618190 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:20.618282 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:20.618359 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:20.618452 (+ 93us) server_negotiation.cc:299] Negotiation successful
0926 03:58:20.618521 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
W20260926 03:58:20.619086 1067 tablet_service.cc:2908] Rejecting scan request for tablet 9331eea4406548ad934f40ce75d343b3: Uninitialized: clean time has not yet been initialized
I20260926 03:58:20.619985 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 1024
I20260926 03:58:20.624948 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 1158
I20260926 03:58:20.631321 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 1292
I20260926 03:58:20.635867 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 930
2026-09-26T03:58:20Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (3312 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:58:22 dist-test-slave-nhpw krb5kdc[1464](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:58:22 dist-test-slave-nhpw krb5kdc[1464](info): set up 2 sockets
Sep 26 03:58:22 dist-test-slave-nhpw krb5kdc[1464](info): commencing operation
krb5kdc: starting...
W20260926 03:58:23.064607 392 mini_kdc.cc:121] Time spent starting KDC: real 2.407s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260926 03:58:23.219362 392 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.155s user 0.000s sys 0.001s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260926 03:58:23.512320 392 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.200s user 0.000s sys 0.001s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/krb5kdc/test-user.keytab.
W20260926 03:58:23.768436 392 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.256s user 0.000s sys 0.002s
Sep 26 03:58:23 dist-test-slave-nhpw krb5kdc[1464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:58:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:58:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab.
W20260926 03:58:24.682636 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.62: real 0.893s user 0.000s sys 0.002s
I20260926 03:58:24.683050 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46765
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46765
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:58:24.770699 1477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:58:24.770918 1477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:58:24.770941 1477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:58:24.772123 1477 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:58:24.772167 1477 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 03:58:24.772181 1477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:58:24.772193 1477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:58:24.772207 1477 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:58:24.772218 1477 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:58:24.773782 1477 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46765
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:58:24.774024 1477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:58:24.774276 1477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:58:24.779030 1485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:24.779163 1483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:24.779030 1482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:24.779279 1477 server_base.cc:1034] running on GCE node
I20260926 03:58:24.779502 1477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:58:24.779788 1477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:58:24.780958 1477 hybrid_clock.cc:648] HybridClock initialized: now 1790395104780938 us; error 36 us; skew 500 ppm
Sep 26 03:58:24 dist-test-slave-nhpw krb5kdc[1464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395104, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:58:24.786983 1477 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 03:58:24.787395 1477 webserver.cc:460] Webserver started at http://127.0.98.62:43229/ using document root <none> and password file <none>
I20260926 03:58:24.787637 1477 fs_manager.cc:359] Metadata directory not provided
I20260926 03:58:24.787705 1477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:58:24.787858 1477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:58:24.789011 1477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/minicluster-data/master-0/data/instance:
uuid: "df5fd5aad81a453790d435063bdd82d4"
format_stamp: "Formatted at 2026-09-26 03:58:24 on dist-test-slave-nhpw"
server_key: "d222db812e562e9f30fc773553a24e98"
server_key_iv: "039b4fe1ca84dfb2828648a25fd2d559"
server_key_version: "encryptionkey@0"
I20260926 03:58:24.789367 1477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/minicluster-data/master-0/wal/instance:
uuid: "df5fd5aad81a453790d435063bdd82d4"
format_stamp: "Formatted at 2026-09-26 03:58:24 on dist-test-slave-nhpw"
server_key: "d222db812e562e9f30fc773553a24e98"
server_key_iv: "039b4fe1ca84dfb2828648a25fd2d559"
server_key_version: "encryptionkey@0"
I20260926 03:58:24.790683 1477 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 03:58:24.791394 1492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:24.791565 1477 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 03:58:24.791656 1477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/minicluster-data/master-0/wal
uuid: "df5fd5aad81a453790d435063bdd82d4"
format_stamp: "Formatted at 2026-09-26 03:58:24 on dist-test-slave-nhpw"
server_key: "d222db812e562e9f30fc773553a24e98"
server_key_iv: "039b4fe1ca84dfb2828648a25fd2d559"
server_key_version: "encryptionkey@0"
I20260926 03:58:24.791738 1477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:58:24.807582 1477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:58:24.809968 1477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:58:24.810140 1477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:58:24.814841 1477 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46765
I20260926 03:58:24.814894 1554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46765 every 8 connection(s)
I20260926 03:58:24.815246 1477 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
I20260926 03:58:24.816006 1555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:58:24.818033 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 1477
I20260926 03:58:24.818128 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/minicluster-data/master-0/wal/instance
I20260926 03:58:24.818295 392 external_mini_cluster.cc:1428] Setting key f808f1ab047c04b51ad65d1f798864b2
I20260926 03:58:24.818591 1555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df5fd5aad81a453790d435063bdd82d4: Bootstrap starting.
I20260926 03:58:24.819541 1555 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P df5fd5aad81a453790d435063bdd82d4: Neither blocks nor log segments found. Creating new log.
I20260926 03:58:24.819880 1555 log.cc:824] T 00000000000000000000000000000000 P df5fd5aad81a453790d435063bdd82d4: Log is configured to *not* fsync() on all Append() calls
I20260926 03:58:24.820785 1555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P df5fd5aad81a453790d435063bdd82d4: No bootstrap required, opened a new log
I20260926 03:58:24.823035 1555 raft_consensus.cc:348] T 00000000000000000000000000000000 P df5fd5aad81a453790d435063bdd82d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df5fd5aad81a453790d435063bdd82d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46765 } }
I20260926 03:58:24.823217 1555 raft_consensus.cc:374] T 00000000000000000000000000000000 P df5fd5aad81a453790d435063bdd82d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:58:24.823249 1555 raft_consensus.cc:729] T 00000000000000000000000000000000 P df5fd5aad81a453790d435063bdd82d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: df5fd5aad81a453790d435063bdd82d4, State: Initialized, Role: FOLLOWER
I20260926 03:58:24.823385 1555 consensus_queue.cc:260] T 00000000000000000000000000000000 P df5fd5aad81a453790d435063bdd82d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df5fd5aad81a453790d435063bdd82d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46765 } }
I20260926 03:58:24.823455 1555 raft_consensus.cc:388] T 00000000000000000000000000000000 P df5fd5aad81a453790d435063bdd82d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:58:24.823485 1555 raft_consensus.cc:482] T 00000000000000000000000000000000 P df5fd5aad81a453790d435063bdd82d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:58:24.823529 1555 raft_consensus.cc:3037] T 00000000000000000000000000000000 P df5fd5aad81a453790d435063bdd82d4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:58:24.824118 1555 raft_consensus.cc:504] T 00000000000000000000000000000000 P df5fd5aad81a453790d435063bdd82d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df5fd5aad81a453790d435063bdd82d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46765 } }
I20260926 03:58:24.824252 1555 leader_election.cc:302] T 00000000000000000000000000000000 P df5fd5aad81a453790d435063bdd82d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: df5fd5aad81a453790d435063bdd82d4; no voters:
I20260926 03:58:24.824457 1555 leader_election.cc:288] T 00000000000000000000000000000000 P df5fd5aad81a453790d435063bdd82d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:58:24.824486 1560 raft_consensus.cc:2781] T 00000000000000000000000000000000 P df5fd5aad81a453790d435063bdd82d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:58:24.824594 1560 raft_consensus.cc:686] T 00000000000000000000000000000000 P df5fd5aad81a453790d435063bdd82d4 [term 1 LEADER]: Becoming Leader. State: Replica: df5fd5aad81a453790d435063bdd82d4, State: Running, Role: LEADER
I20260926 03:58:24.824752 1555 sys_catalog.cc:564] T 00000000000000000000000000000000 P df5fd5aad81a453790d435063bdd82d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:58:24.824780 1560 consensus_queue.cc:237] T 00000000000000000000000000000000 P df5fd5aad81a453790d435063bdd82d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df5fd5aad81a453790d435063bdd82d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46765 } }
I20260926 03:58:24.825135 1561 sys_catalog.cc:455] T 00000000000000000000000000000000 P df5fd5aad81a453790d435063bdd82d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "df5fd5aad81a453790d435063bdd82d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df5fd5aad81a453790d435063bdd82d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46765 } } }
I20260926 03:58:24.825243 1561 sys_catalog.cc:458] T 00000000000000000000000000000000 P df5fd5aad81a453790d435063bdd82d4 [sys.catalog]: This master's current role is: LEADER
I20260926 03:58:24.825124 1560 sys_catalog.cc:455] T 00000000000000000000000000000000 P df5fd5aad81a453790d435063bdd82d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader df5fd5aad81a453790d435063bdd82d4. Latest consensus state: current_term: 1 leader_uuid: "df5fd5aad81a453790d435063bdd82d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "df5fd5aad81a453790d435063bdd82d4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46765 } } }
I20260926 03:58:24.825425 1560 sys_catalog.cc:458] T 00000000000000000000000000000000 P df5fd5aad81a453790d435063bdd82d4 [sys.catalog]: This master's current role is: LEADER
I20260926 03:58:24.825551 1567 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 26 03:58:24 dist-test-slave-nhpw krb5kdc[1464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 03:58:24.826097 1567 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:58:24.827701 1567 catalog_manager.cc:1347] Generated new cluster ID: 94fe7cdf447b41feb3ff153a3983849c
I20260926 03:58:24.827791 1567 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:58:24.830384 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:24.818789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54462 (local address 127.0.98.62:46765)
0926 03:58:24.818943 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:24.818953 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:24.818984 (+ 31us) server_negotiation.cc:407] Connection header received
0926 03:58:24.819623 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:24.819631 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:24.819705 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:24.819868 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:24.820794 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:24.822243 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:24.822968 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:24.823569 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:24.823679 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:24.826732 (+ 3053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:24.826737 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:24.826737 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:24.826751 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:24.828788 (+ 2037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:24.829209 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:24.829212 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:24.829213 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:24.829306 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:24.829525 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:24.829525 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:24.829526 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:24.829800 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:24.829935 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:24.830062 (+ 127us) server_negotiation.cc:299] Negotiation successful
0926 03:58:24.830183 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":84,"threads_started":1}
I20260926 03:58:24.857589 1567 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:58:24.858160 1567 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:58:24.863265 1567 catalog_manager.cc:5868] T 00000000000000000000000000000000 P df5fd5aad81a453790d435063bdd82d4: Generated new TSK 0
I20260926 03:58:24.863459 1567 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 03:58:24.896421 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46765
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:58:24.978710 1581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:58:24.978956 1581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:58:24.978998 1581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:58:24.980165 1581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:58:24.980233 1581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:58:24.980273 1581 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 03:58:24.980309 1581 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 03:58:24.981870 1581 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:58:24.982137 1581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:58:24.982376 1581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:58:24.987428 1587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:24.987488 1586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:24.987427 1589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:24.987625 1581 server_base.cc:1034] running on GCE node
I20260926 03:58:24.987840 1581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:58:24.988041 1581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:58:24.989180 1581 hybrid_clock.cc:648] HybridClock initialized: now 1790395104989167 us; error 26 us; skew 500 ppm
Sep 26 03:58:24 dist-test-slave-nhpw krb5kdc[1464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395104, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:58:24.991633 1581 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 03:58:24.992070 1581 webserver.cc:460] Webserver started at http://127.0.98.1:32905/ using document root <none> and password file <none>
I20260926 03:58:24.992260 1581 fs_manager.cc:359] Metadata directory not provided
I20260926 03:58:24.992300 1581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:58:24.992388 1581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:58:24.993449 1581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/minicluster-data/ts-0/data/instance:
uuid: "266532c1f533453fb6df898015566b66"
format_stamp: "Formatted at 2026-09-26 03:58:24 on dist-test-slave-nhpw"
server_key: "ccd66db0ed0eacfec9f8ffd4095d2ff0"
server_key_iv: "aefa8dc2a9af436f8df15669ad6fe4f8"
server_key_version: "encryptionkey@0"
I20260926 03:58:24.993747 1581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/minicluster-data/ts-0/wal/instance:
uuid: "266532c1f533453fb6df898015566b66"
format_stamp: "Formatted at 2026-09-26 03:58:24 on dist-test-slave-nhpw"
server_key: "ccd66db0ed0eacfec9f8ffd4095d2ff0"
server_key_iv: "aefa8dc2a9af436f8df15669ad6fe4f8"
server_key_version: "encryptionkey@0"
I20260926 03:58:24.995069 1581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 03:58:24.995781 1596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:24.995999 1581 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 03:58:24.996070 1581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/minicluster-data/ts-0/wal
uuid: "266532c1f533453fb6df898015566b66"
format_stamp: "Formatted at 2026-09-26 03:58:24 on dist-test-slave-nhpw"
server_key: "ccd66db0ed0eacfec9f8ffd4095d2ff0"
server_key_iv: "aefa8dc2a9af436f8df15669ad6fe4f8"
server_key_version: "encryptionkey@0"
I20260926 03:58:24.996160 1581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:58:25.009112 1581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:58:25.011917 1581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:58:25.012063 1581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:58:25.012595 1581 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:58:25.012681 1581 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:25.012727 1581 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:58:25.012753 1581 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 03:58:25 dist-test-slave-nhpw krb5kdc[1464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395104, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 03:58:25.019694 1581 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44309
I20260926 03:58:25.019763 1711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44309 every 8 connection(s)
I20260926 03:58:25.020066 1581 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
I20260926 03:58:25.021462 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 1581
I20260926 03:58:25.021564 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/minicluster-data/ts-0/wal/instance
I20260926 03:58:25.021749 392 external_mini_cluster.cc:1428] Setting key e6fc479ac72486d4e3d2d5fe237705da
I20260926 03:58:25.022920 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:25.013135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52615 (local address 127.0.98.62:46765)
0926 03:58:25.013177 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:25.013179 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:25.013380 (+ 201us) server_negotiation.cc:407] Connection header received
0926 03:58:25.013963 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:25.013965 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:25.013978 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:25.014035 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:25.014899 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:25.015743 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:25.017158 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:25.017456 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:25.017542 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:25.019716 (+ 2174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:25.019718 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:25.019719 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:25.019727 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:25.021298 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:25.021781 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:25.021782 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:25.021783 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:25.021899 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:25.022213 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:25.022214 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:25.022215 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:25.022375 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:25.022451 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:25.022769 (+ 318us) server_negotiation.cc:299] Negotiation successful
0926 03:58:25.022856 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 03:58:25.023211 1605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:25.013124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46765 (local address 127.0.98.1:52615)
0926 03:58:25.013264 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:25.013297 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:25.013817 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:25.014112 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:25.014117 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:25.014501 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:25.014779 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:25.014784 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:25.015839 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:25.015840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:25.017067 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:25.017069 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:25.017561 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:25.017562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:25.017692 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:25.018209 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:25.018217 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:25.019621 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:58:25.021382 (+ 1761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:25.021388 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:25.021390 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:25.021707 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:25.022008 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:25.022009 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:25.022009 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:25.022144 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:25.022454 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:58:25.022460 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:58:25.022625 (+ 165us) client_negotiation.cc:769] Sending connection context
0926 03:58:25.022742 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 03:58:25.022933 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":79,"threads_started":1}
I20260926 03:58:25.023698 1712 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46765
I20260926 03:58:25.023816 1712 heartbeater.cc:464] Registering TS with master...
I20260926 03:58:25.023995 1712 heartbeater.cc:511] Master 127.0.98.62:46765 requested a full tablet report, sending...
I20260926 03:58:25.024879 1508 ts_manager.cc:194] Registered new tserver with Master: 266532c1f533453fb6df898015566b66 (127.0.98.1:44309)
I20260926 03:58:25.025596 1508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52615
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 03:58:25.031374 1719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:25.024720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46765 (local address 127.0.98.1:42819)
0926 03:58:25.025831 (+ 1111us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:25.025840 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:25.025923 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:25.026052 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:25.026052 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:25.026236 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:25.026288 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:25.026289 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:25.026991 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:25.026992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:25.027529 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:25.027530 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:25.027902 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:25.027903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:25.027971 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:25.028407 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:25.028412 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:25.028753 (+ 341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:58:25.030356 (+ 1603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:25.030357 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:25.030357 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:25.030524 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:25.030735 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:25.030736 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:25.030737 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:25.030793 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:25.031044 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:58:25.031044 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:58:25.031086 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 03:58:25.031173 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 03:58:25.031271 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1089,"thread_start_us":59,"threads_started":1}
I20260926 03:58:25.031363 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:25.024768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42819 (local address 127.0.98.62:46765)
0926 03:58:25.024806 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:25.024808 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:25.025909 (+ 1101us) server_negotiation.cc:407] Connection header received
0926 03:58:25.025981 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:25.025981 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:25.025991 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:25.026046 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:25.026371 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:25.026902 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:25.027623 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:25.027820 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:25.027897 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:25.028851 (+ 954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:25.028852 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:25.028852 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:25.028859 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:25.030254 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:25.030610 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:25.030611 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:25.030612 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:25.030662 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:25.030866 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:25.030867 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:25.030868 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:25.030976 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:25.031051 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:25.031198 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 03:58:25.031271 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 03:58:25.047616 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46765
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:58:25.128952 1721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:58:25.129148 1721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:58:25.129171 1721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:58:25.130317 1721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:58:25.130359 1721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:58:25.130380 1721 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 03:58:25.130427 1721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 03:58:25.132014 1721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:58:25.132256 1721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:58:25.132494 1721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:58:25.137454 1727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:25.137445 1726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:25.137610 1721 server_base.cc:1034] running on GCE node
W20260926 03:58:25.137442 1729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:25.137976 1721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:58:25.138221 1721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:58:25.139365 1721 hybrid_clock.cc:648] HybridClock initialized: now 1790395105139351 us; error 27 us; skew 500 ppm
Sep 26 03:58:25 dist-test-slave-nhpw krb5kdc[1464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395105, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:58:25.145776 1721 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 03:58:25.146272 1721 webserver.cc:460] Webserver started at http://127.0.98.2:35859/ using document root <none> and password file <none>
I20260926 03:58:25.146509 1721 fs_manager.cc:359] Metadata directory not provided
I20260926 03:58:25.146579 1721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:58:25.146735 1721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:58:25.147768 1721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/minicluster-data/ts-1/data/instance:
uuid: "113a195864e74a88a6b10daacba06759"
format_stamp: "Formatted at 2026-09-26 03:58:25 on dist-test-slave-nhpw"
server_key: "b186c2b72a87b67bd1ecd73fe54b5c4a"
server_key_iv: "2997867ffa58a487141700ab34c9607a"
server_key_version: "encryptionkey@0"
I20260926 03:58:25.148123 1721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/minicluster-data/ts-1/wal/instance:
uuid: "113a195864e74a88a6b10daacba06759"
format_stamp: "Formatted at 2026-09-26 03:58:25 on dist-test-slave-nhpw"
server_key: "b186c2b72a87b67bd1ecd73fe54b5c4a"
server_key_iv: "2997867ffa58a487141700ab34c9607a"
server_key_version: "encryptionkey@0"
I20260926 03:58:25.149513 1721 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 03:58:25.150276 1736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:25.150461 1721 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 03:58:25.150563 1721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/minicluster-data/ts-1/wal
uuid: "113a195864e74a88a6b10daacba06759"
format_stamp: "Formatted at 2026-09-26 03:58:25 on dist-test-slave-nhpw"
server_key: "b186c2b72a87b67bd1ecd73fe54b5c4a"
server_key_iv: "2997867ffa58a487141700ab34c9607a"
server_key_version: "encryptionkey@0"
I20260926 03:58:25.150633 1721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:58:25.184702 1721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:58:25.186684 1721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:58:25.186865 1721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:58:25.187448 1721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:58:25.187510 1721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:25.187553 1721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:58:25.187575 1721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 03:58:25 dist-test-slave-nhpw krb5kdc[1464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395105, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 03:58:25.194841 1721 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45311
I20260926 03:58:25.194919 1851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45311 every 8 connection(s)
I20260926 03:58:25.195232 1721 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
I20260926 03:58:25.197710 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:25.187911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53509 (local address 127.0.98.62:46765)
0926 03:58:25.187973 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:25.187976 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:25.188174 (+ 198us) server_negotiation.cc:407] Connection header received
0926 03:58:25.188666 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:25.188668 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:25.188681 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:25.188761 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:25.189342 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:25.190190 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:25.191240 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:25.191556 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:25.191641 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:25.194943 (+ 3302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:25.194945 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:25.194945 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:25.194953 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:25.196533 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:25.196962 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:25.196963 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:25.196963 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:25.197015 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:25.197225 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:25.197225 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:25.197226 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:25.197326 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:25.197376 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:25.197592 (+ 216us) server_negotiation.cc:299] Negotiation successful
0926 03:58:25.197649 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 03:58:25.197958 1745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:25.187867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46765 (local address 127.0.98.2:53509)
0926 03:58:25.188032 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:25.188074 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:25.188567 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:25.188816 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:25.188819 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:25.189048 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:25.189229 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:25.189234 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:25.190288 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:25.190289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:25.191127 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:25.191129 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:25.191651 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:25.191652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:25.191786 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:25.192300 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:25.192313 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:25.194805 (+ 2492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:58:25.196606 (+ 1801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:25.196610 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:25.196611 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:25.196880 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:25.197078 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:25.197079 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:25.197079 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:25.197173 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:25.197380 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:58:25.197384 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:58:25.197493 (+ 109us) client_negotiation.cc:769] Sending connection context
0926 03:58:25.197574 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 03:58:25.197688 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":100,"threads_started":1}
I20260926 03:58:25.198483 1852 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46765
I20260926 03:58:25.198565 1852 heartbeater.cc:464] Registering TS with master...
I20260926 03:58:25.198738 1852 heartbeater.cc:511] Master 127.0.98.62:46765 requested a full tablet report, sending...
I20260926 03:58:25.199153 1509 ts_manager.cc:194] Registered new tserver with Master: 113a195864e74a88a6b10daacba06759 (127.0.98.2:45311)
I20260926 03:58:25.199646 1509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53509
I20260926 03:58:25.202637 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 1721
I20260926 03:58:25.202754 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/minicluster-data/ts-1/wal/instance
I20260926 03:58:25.202934 392 external_mini_cluster.cc:1428] Setting key 9bace89d00ad9c51fbc6fd15cf617660
I20260926 03:58:25.205253 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:25.199606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41865 (local address 127.0.98.62:46765)
0926 03:58:25.199647 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:25.199649 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:25.199744 (+ 95us) server_negotiation.cc:407] Connection header received
0926 03:58:25.199878 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:25.199880 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:25.199890 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:25.199950 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:25.200238 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:25.200828 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:25.201485 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:25.201653 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:25.201702 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:25.202403 (+ 701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:25.202404 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:25.202404 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:25.202410 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:25.204089 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:25.204391 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:25.204392 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:25.204392 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:25.204458 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:25.204629 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:25.204629 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:25.204630 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:25.204843 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:25.204915 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:25.205109 (+ 194us) server_negotiation.cc:299] Negotiation successful
0926 03:58:25.205188 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 03:58:25.205300 1858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:25.199531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46765 (local address 127.0.98.2:41865)
0926 03:58:25.199663 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:25.199676 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:25.199738 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:25.199951 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:25.199952 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:25.200117 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:25.200165 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:25.200166 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:25.200913 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:25.200913 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:25.201418 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:25.201419 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:25.201716 (+ 297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:25.201716 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:25.201756 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:25.202083 (+ 327us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:25.202087 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:25.202330 (+ 243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:58:25.204168 (+ 1838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:25.204169 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:25.204169 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:25.204320 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:25.204522 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:25.204523 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:25.204523 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:25.204570 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:25.204928 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:58:25.204929 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:58:25.204977 (+ 48us) client_negotiation.cc:769] Sending connection context
0926 03:58:25.205064 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 03:58:25.205171 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":80,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 03:58:25.228559 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46765
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:58:25.323253 1861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:58:25.323459 1861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:58:25.323484 1861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:58:25.324959 1861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:58:25.325006 1861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:58:25.325052 1861 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 03:58:25.325078 1861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 03:58:25.326748 1861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46765
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1861
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:58:25.326930 1861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:58:25.327103 1861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:58:25.331944 1867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:25.331987 1869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:25.331950 1866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:25.332157 1861 server_base.cc:1034] running on GCE node
I20260926 03:58:25.332340 1861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:58:25.332613 1861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:58:25.333838 1861 hybrid_clock.cc:648] HybridClock initialized: now 1790395105333828 us; error 31 us; skew 500 ppm
Sep 26 03:58:25 dist-test-slave-nhpw krb5kdc[1464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395105, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:58:25.336344 1861 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 03:58:25.336840 1861 webserver.cc:460] Webserver started at http://127.0.98.3:39289/ using document root <none> and password file <none>
I20260926 03:58:25.337026 1861 fs_manager.cc:359] Metadata directory not provided
I20260926 03:58:25.337066 1861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:58:25.337154 1861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:58:25.338126 1861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/minicluster-data/ts-2/data/instance:
uuid: "55485c830e8a45e68078486b3b60ec47"
format_stamp: "Formatted at 2026-09-26 03:58:25 on dist-test-slave-nhpw"
server_key: "692f85a301cd7da09ebf7979f1affbd7"
server_key_iv: "dbc68bd397701b468784ee9cce04da60"
server_key_version: "encryptionkey@0"
I20260926 03:58:25.338421 1861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/minicluster-data/ts-2/wal/instance:
uuid: "55485c830e8a45e68078486b3b60ec47"
format_stamp: "Formatted at 2026-09-26 03:58:25 on dist-test-slave-nhpw"
server_key: "692f85a301cd7da09ebf7979f1affbd7"
server_key_iv: "dbc68bd397701b468784ee9cce04da60"
server_key_version: "encryptionkey@0"
I20260926 03:58:25.339660 1861 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 03:58:25.340377 1876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:25.340556 1861 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 03:58:25.340672 1861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/minicluster-data/ts-2/wal
uuid: "55485c830e8a45e68078486b3b60ec47"
format_stamp: "Formatted at 2026-09-26 03:58:25 on dist-test-slave-nhpw"
server_key: "692f85a301cd7da09ebf7979f1affbd7"
server_key_iv: "dbc68bd397701b468784ee9cce04da60"
server_key_version: "encryptionkey@0"
I20260926 03:58:25.340759 1861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:58:25.358594 1861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:58:25.360940 1861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:58:25.361126 1861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:58:25.361753 1861 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:58:25.361804 1861 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:25.361833 1861 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:58:25.361856 1861 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 03:58:25 dist-test-slave-nhpw krb5kdc[1464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395105, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 03:58:25.369005 1861 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38109
I20260926 03:58:25.369082 1991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38109 every 8 connection(s)
I20260926 03:58:25.369387 1861 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
I20260926 03:58:25.371595 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:25.362127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51811 (local address 127.0.98.62:46765)
0926 03:58:25.362186 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:25.362189 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:25.362443 (+ 254us) server_negotiation.cc:407] Connection header received
0926 03:58:25.362925 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:25.362926 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:25.362939 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:25.363048 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:25.363830 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:25.364705 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:25.365758 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:25.365970 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:25.366058 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:25.368730 (+ 2672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:25.368732 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:25.368733 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:25.368743 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:25.370383 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:25.370750 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:25.370750 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:25.370750 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:25.370800 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:25.371006 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:25.371006 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:25.371006 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:25.371114 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:25.371167 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:25.371476 (+ 309us) server_negotiation.cc:299] Negotiation successful
0926 03:58:25.371535 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 03:58:25.371791 1885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:25.362139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46765 (local address 127.0.98.3:51811)
0926 03:58:25.362303 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:25.362342 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:25.362814 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:25.363145 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:25.363150 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:25.363526 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:25.363741 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:25.363748 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:25.364813 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:25.364814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:25.365645 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:25.365648 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:25.366067 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:25.366068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:25.366197 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:25.366813 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:25.366821 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:25.368587 (+ 1766us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:58:25.370472 (+ 1885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:25.370476 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:25.370478 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:25.370683 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:25.370857 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:25.370857 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:25.370857 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:25.370947 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:25.371230 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:58:25.371236 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:58:25.371355 (+ 119us) client_negotiation.cc:769] Sending connection context
0926 03:58:25.371454 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 03:58:25.371573 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":96,"threads_started":1}
I20260926 03:58:25.372265 1992 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46765
I20260926 03:58:25.372349 1992 heartbeater.cc:464] Registering TS with master...
I20260926 03:58:25.372534 1992 heartbeater.cc:511] Master 127.0.98.62:46765 requested a full tablet report, sending...
I20260926 03:58:25.372952 1509 ts_manager.cc:194] Registered new tserver with Master: 55485c830e8a45e68078486b3b60ec47 (127.0.98.3:38109)
I20260926 03:58:25.373200 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 1861
I20260926 03:58:25.373278 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/minicluster-data/ts-2/wal/instance
I20260926 03:58:25.373454 392 external_mini_cluster.cc:1428] Setting key 4305af892be7578ab4955353db85d1fd
I20260926 03:58:25.373441 1509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51811
I20260926 03:58:25.374617 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:58:25.379004 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:25.373339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33305 (local address 127.0.98.62:46765)
0926 03:58:25.373376 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:25.373378 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:25.373685 (+ 307us) server_negotiation.cc:407] Connection header received
0926 03:58:25.373704 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:25.373704 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:25.373714 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:25.373760 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:25.374087 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:25.374557 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:25.375300 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:25.375510 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:25.375594 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:25.376474 (+ 880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:25.376475 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:25.376476 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:25.376483 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:25.377891 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:25.378255 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:25.378256 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:25.378257 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:25.378308 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:25.378510 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:25.378511 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:25.378511 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:25.378618 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:25.378693 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:25.378843 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 03:58:25.378915 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 03:58:25.379019 1998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:25.373385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46765 (local address 127.0.98.3:33305)
0926 03:58:25.373582 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:25.373593 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:25.373650 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:25.373762 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:25.373763 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:25.373947 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:25.374000 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:25.374002 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:25.374652 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:25.374653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:25.375199 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:25.375200 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:25.375595 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:25.375596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:25.375659 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:25.376118 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:25.376122 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:25.376381 (+ 259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:58:25.377997 (+ 1616us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:25.377998 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:25.377998 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:25.378168 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:25.378382 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:25.378383 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:25.378383 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:25.378446 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:25.378685 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:58:25.378686 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:58:25.378727 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 03:58:25.378818 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 03:58:25.378924 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":102,"threads_started":1}
Sep 26 03:58:25 dist-test-slave-nhpw krb5kdc[1464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395105, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 03:58:25 dist-test-slave-nhpw krb5kdc[1464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395105, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 03:58:25.397061 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:25.390053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54472 (local address 127.0.98.62:46765)
0926 03:58:25.390114 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:25.390115 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:25.390124 (+ 9us) server_negotiation.cc:407] Connection header received
0926 03:58:25.390183 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:25.390184 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:25.390193 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:25.390258 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:25.390779 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:25.391274 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:25.391977 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:25.392162 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:25.392218 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:25.394515 (+ 2297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:25.394517 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:25.394517 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:25.394523 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:25.396040 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:25.396374 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:25.396375 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:25.396375 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:25.396422 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:25.396595 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:25.396595 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:25.396595 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:25.396748 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:25.396820 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:25.396942 (+ 122us) server_negotiation.cc:299] Negotiation successful
0926 03:58:25.396998 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 03:58:25.398902 1509 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54472:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:58:25.400694 1509 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:58:25.407470 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:25.403266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44309 (local address 127.0.0.1:58220)
0926 03:58:25.403389 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:25.403399 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:25.403442 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:25.403750 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:25.403753 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:25.403755 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:58:25.403852 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:25.403853 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:25.405050 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:25.405051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:25.406326 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:25.406328 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:25.407219 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:25.407221 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:25.407287 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:25.407292 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 03:58:25.407323 (+ 31us) client_negotiation.cc:240] Negotiation successful
0926 03:58:25.407389 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":74,"threads_started":1}
I20260926 03:58:25.407512 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:25.403489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38124 (local address 127.0.98.3:38109)
0926 03:58:25.403666 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:25.403671 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:25.403682 (+ 11us) server_negotiation.cc:407] Connection header received
0926 03:58:25.403723 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:25.403725 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:25.403741 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:25.403833 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:58:25.403940 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:25.405156 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:25.406554 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:25.407033 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:25.407124 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:25.407203 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:25.407366 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 03:58:25.407409 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":90,"threads_started":1}
I20260926 03:58:25.407567 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:25.403502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52074 (local address 127.0.98.2:45311)
0926 03:58:25.403737 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:25.403740 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:25.403749 (+ 9us) server_negotiation.cc:407] Connection header received
0926 03:58:25.403784 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:25.403786 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:25.403801 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:25.403880 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:58:25.404078 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:25.405268 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:25.406545 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:25.407238 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:25.407303 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:25.407377 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:25.407435 (+ 58us) server_negotiation.cc:299] Negotiation successful
0926 03:58:25.407470 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":113,"threads_started":1}
I20260926 03:58:25.407780 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:25.403047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38109 (local address 127.0.0.1:38124)
0926 03:58:25.403201 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:25.403211 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:25.403258 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:25.403839 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:25.403840 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:25.403841 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:58:25.403879 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:25.403880 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:25.405255 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:25.405256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:25.406459 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:25.406461 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:25.407135 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:25.407136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:25.407218 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:25.407232 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 03:58:25.407297 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 03:58:25.407709 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":367,"thread_start_us":109,"threads_started":1}
I20260926 03:58:25.407989 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:25.402971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45311 (local address 127.0.0.1:52074)
0926 03:58:25.403068 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:25.403112 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:25.403174 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:25.403947 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:25.403948 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:25.403950 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:58:25.403999 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:25.404001 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:25.405352 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:25.405352 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:25.406444 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:25.406447 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:25.407304 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:25.407305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:25.407368 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:25.407373 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 03:58:25.407426 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 03:58:25.407460 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74,"mutex_wait_us":4,"thread_start_us":58,"threads_started":1}
I20260926 03:58:25.408197 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:25.403358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58220 (local address 127.0.98.1:44309)
0926 03:58:25.403562 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:25.403566 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:25.403578 (+ 12us) server_negotiation.cc:407] Connection header received
0926 03:58:25.403619 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:25.403621 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:25.403636 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:25.403711 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:58:25.403964 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:25.404923 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:25.406412 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:25.407117 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:25.407921 (+ 804us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:25.408002 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:25.408028 (+ 26us) server_negotiation.cc:299] Negotiation successful
0926 03:58:25.408092 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":102,"threads_started":1}
I20260926 03:58:25.408474 1926 tablet_service.cc:1467] Processing CreateTablet for tablet b522d5900f164785827e7d8a801e38e0 (DEFAULT_TABLE table=test-table [id=44e35391cd724a10a3672843c27c6d4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:58:25.408555 1786 tablet_service.cc:1467] Processing CreateTablet for tablet b522d5900f164785827e7d8a801e38e0 (DEFAULT_TABLE table=test-table [id=44e35391cd724a10a3672843c27c6d4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:58:25.408885 1786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b522d5900f164785827e7d8a801e38e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:58:25.408885 1926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b522d5900f164785827e7d8a801e38e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:58:25.408864 1645 tablet_service.cc:1467] Processing CreateTablet for tablet b522d5900f164785827e7d8a801e38e0 (DEFAULT_TABLE table=test-table [id=44e35391cd724a10a3672843c27c6d4c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:58:25.409189 1645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b522d5900f164785827e7d8a801e38e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:58:25.412123 2015 tablet_bootstrap.cc:492] T b522d5900f164785827e7d8a801e38e0 P 266532c1f533453fb6df898015566b66: Bootstrap starting.
I20260926 03:58:25.412757 2017 tablet_bootstrap.cc:492] T b522d5900f164785827e7d8a801e38e0 P 113a195864e74a88a6b10daacba06759: Bootstrap starting.
I20260926 03:58:25.412892 2015 tablet_bootstrap.cc:654] T b522d5900f164785827e7d8a801e38e0 P 266532c1f533453fb6df898015566b66: Neither blocks nor log segments found. Creating new log.
I20260926 03:58:25.413158 2015 log.cc:824] T b522d5900f164785827e7d8a801e38e0 P 266532c1f533453fb6df898015566b66: Log is configured to *not* fsync() on all Append() calls
I20260926 03:58:25.413348 2017 tablet_bootstrap.cc:654] T b522d5900f164785827e7d8a801e38e0 P 113a195864e74a88a6b10daacba06759: Neither blocks nor log segments found. Creating new log.
I20260926 03:58:25.413640 2017 log.cc:824] T b522d5900f164785827e7d8a801e38e0 P 113a195864e74a88a6b10daacba06759: Log is configured to *not* fsync() on all Append() calls
I20260926 03:58:25.413862 2015 tablet_bootstrap.cc:492] T b522d5900f164785827e7d8a801e38e0 P 266532c1f533453fb6df898015566b66: No bootstrap required, opened a new log
I20260926 03:58:25.413944 2015 ts_tablet_manager.cc:1397] T b522d5900f164785827e7d8a801e38e0 P 266532c1f533453fb6df898015566b66: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 03:58:25.413956 2016 tablet_bootstrap.cc:492] T b522d5900f164785827e7d8a801e38e0 P 55485c830e8a45e68078486b3b60ec47: Bootstrap starting.
I20260926 03:58:25.414306 2017 tablet_bootstrap.cc:492] T b522d5900f164785827e7d8a801e38e0 P 113a195864e74a88a6b10daacba06759: No bootstrap required, opened a new log
I20260926 03:58:25.414364 2017 ts_tablet_manager.cc:1397] T b522d5900f164785827e7d8a801e38e0 P 113a195864e74a88a6b10daacba06759: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 03:58:25.415627 2015 raft_consensus.cc:348] T b522d5900f164785827e7d8a801e38e0 P 266532c1f533453fb6df898015566b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "113a195864e74a88a6b10daacba06759" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45311 } } peers { permanent_uuid: "55485c830e8a45e68078486b3b60ec47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38109 } } peers { permanent_uuid: "266532c1f533453fb6df898015566b66" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44309 } }
I20260926 03:58:25.415795 2015 raft_consensus.cc:374] T b522d5900f164785827e7d8a801e38e0 P 266532c1f533453fb6df898015566b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:58:25.415851 2015 raft_consensus.cc:729] T b522d5900f164785827e7d8a801e38e0 P 266532c1f533453fb6df898015566b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 266532c1f533453fb6df898015566b66, State: Initialized, Role: FOLLOWER
I20260926 03:58:25.415757 2017 raft_consensus.cc:348] T b522d5900f164785827e7d8a801e38e0 P 113a195864e74a88a6b10daacba06759 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "113a195864e74a88a6b10daacba06759" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45311 } } peers { permanent_uuid: "55485c830e8a45e68078486b3b60ec47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38109 } } peers { permanent_uuid: "266532c1f533453fb6df898015566b66" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44309 } }
I20260926 03:58:25.415923 2017 raft_consensus.cc:374] T b522d5900f164785827e7d8a801e38e0 P 113a195864e74a88a6b10daacba06759 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:58:25.415968 2017 raft_consensus.cc:729] T b522d5900f164785827e7d8a801e38e0 P 113a195864e74a88a6b10daacba06759 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 113a195864e74a88a6b10daacba06759, State: Initialized, Role: FOLLOWER
I20260926 03:58:25.415972 2015 consensus_queue.cc:260] T b522d5900f164785827e7d8a801e38e0 P 266532c1f533453fb6df898015566b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "113a195864e74a88a6b10daacba06759" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45311 } } peers { permanent_uuid: "55485c830e8a45e68078486b3b60ec47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38109 } } peers { permanent_uuid: "266532c1f533453fb6df898015566b66" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44309 } }
I20260926 03:58:25.416090 2017 consensus_queue.cc:260] T b522d5900f164785827e7d8a801e38e0 P 113a195864e74a88a6b10daacba06759 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "113a195864e74a88a6b10daacba06759" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45311 } } peers { permanent_uuid: "55485c830e8a45e68078486b3b60ec47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38109 } } peers { permanent_uuid: "266532c1f533453fb6df898015566b66" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44309 } }
I20260926 03:58:25.416236 2015 ts_tablet_manager.cc:1428] T b522d5900f164785827e7d8a801e38e0 P 266532c1f533453fb6df898015566b66: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 03:58:25.416262 1712 heartbeater.cc:503] Master 127.0.98.62:46765 was elected leader, sending a full tablet report...
I20260926 03:58:25.416666 2017 ts_tablet_manager.cc:1428] T b522d5900f164785827e7d8a801e38e0 P 113a195864e74a88a6b10daacba06759: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 03:58:25.416690 1852 heartbeater.cc:503] Master 127.0.98.62:46765 was elected leader, sending a full tablet report...
I20260926 03:58:25.417253 2016 tablet_bootstrap.cc:654] T b522d5900f164785827e7d8a801e38e0 P 55485c830e8a45e68078486b3b60ec47: Neither blocks nor log segments found. Creating new log.
I20260926 03:58:25.417614 2016 log.cc:824] T b522d5900f164785827e7d8a801e38e0 P 55485c830e8a45e68078486b3b60ec47: Log is configured to *not* fsync() on all Append() calls
I20260926 03:58:25.418251 2016 tablet_bootstrap.cc:492] T b522d5900f164785827e7d8a801e38e0 P 55485c830e8a45e68078486b3b60ec47: No bootstrap required, opened a new log
I20260926 03:58:25.418301 2016 ts_tablet_manager.cc:1397] T b522d5900f164785827e7d8a801e38e0 P 55485c830e8a45e68078486b3b60ec47: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260926 03:58:25.419598 2016 raft_consensus.cc:348] T b522d5900f164785827e7d8a801e38e0 P 55485c830e8a45e68078486b3b60ec47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "113a195864e74a88a6b10daacba06759" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45311 } } peers { permanent_uuid: "55485c830e8a45e68078486b3b60ec47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38109 } } peers { permanent_uuid: "266532c1f533453fb6df898015566b66" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44309 } }
I20260926 03:58:25.419718 2016 raft_consensus.cc:374] T b522d5900f164785827e7d8a801e38e0 P 55485c830e8a45e68078486b3b60ec47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:58:25.419739 2016 raft_consensus.cc:729] T b522d5900f164785827e7d8a801e38e0 P 55485c830e8a45e68078486b3b60ec47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55485c830e8a45e68078486b3b60ec47, State: Initialized, Role: FOLLOWER
I20260926 03:58:25.419899 2016 consensus_queue.cc:260] T b522d5900f164785827e7d8a801e38e0 P 55485c830e8a45e68078486b3b60ec47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "113a195864e74a88a6b10daacba06759" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45311 } } peers { permanent_uuid: "55485c830e8a45e68078486b3b60ec47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38109 } } peers { permanent_uuid: "266532c1f533453fb6df898015566b66" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44309 } }
I20260926 03:58:25.420166 2016 ts_tablet_manager.cc:1428] T b522d5900f164785827e7d8a801e38e0 P 55485c830e8a45e68078486b3b60ec47: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 03:58:25.420183 1992 heartbeater.cc:503] Master 127.0.98.62:46765 was elected leader, sending a full tablet report...
W20260926 03:58:25.445895 1853 tablet.cc:2369] T b522d5900f164785827e7d8a801e38e0 P 113a195864e74a88a6b10daacba06759: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:58:25.520781 1713 tablet.cc:2369] T b522d5900f164785827e7d8a801e38e0 P 266532c1f533453fb6df898015566b66: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:58:25.527663 2021 raft_consensus.cc:482] T b522d5900f164785827e7d8a801e38e0 P 113a195864e74a88a6b10daacba06759 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:58:25.527776 2021 raft_consensus.cc:504] T b522d5900f164785827e7d8a801e38e0 P 113a195864e74a88a6b10daacba06759 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "113a195864e74a88a6b10daacba06759" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45311 } } peers { permanent_uuid: "55485c830e8a45e68078486b3b60ec47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38109 } } peers { permanent_uuid: "266532c1f533453fb6df898015566b66" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44309 } }
I20260926 03:58:25.528101 2021 leader_election.cc:288] T b522d5900f164785827e7d8a801e38e0 P 113a195864e74a88a6b10daacba06759 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 55485c830e8a45e68078486b3b60ec47 (127.0.98.3:38109), 266532c1f533453fb6df898015566b66 (127.0.98.1:44309)
I20260926 03:58:25.532084 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:25.528413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44087 (local address 127.0.98.3:38109)
0926 03:58:25.528464 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:25.528466 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:25.528477 (+ 11us) server_negotiation.cc:407] Connection header received
0926 03:58:25.528545 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:25.528545 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:25.528557 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:25.528610 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:58:25.528795 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:25.529593 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:25.531147 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:25.531697 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:25.531774 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:25.531784 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:25.531980 (+ 196us) server_negotiation.cc:299] Negotiation successful
0926 03:58:25.532016 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 03:58:25.532084 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:25.528595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39133 (local address 127.0.98.1:44309)
0926 03:58:25.528666 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:25.528668 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:25.528720 (+ 52us) server_negotiation.cc:407] Connection header received
0926 03:58:25.528791 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:25.528792 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:25.528803 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:25.528861 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:58:25.528992 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:25.529685 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:25.531148 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:25.531698 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:25.531774 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:25.531784 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:25.531980 (+ 196us) server_negotiation.cc:299] Negotiation successful
0926 03:58:25.532016 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 03:58:25.532140 1745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:25.528318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38109 (local address 127.0.98.2:44087)
0926 03:58:25.528395 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:25.528409 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:25.528473 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:25.528648 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:25.528649 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:25.528651 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:58:25.528724 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:25.528726 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:25.529709 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:25.529710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:25.531040 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:25.531043 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:25.531799 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:25.531800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:25.531892 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:25.531898 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 03:58:25.531963 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 03:58:25.532012 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":77}
I20260926 03:58:25.532377 2024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:25.528435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44309 (local address 127.0.98.2:39133)
0926 03:58:25.528616 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:25.528626 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:25.528715 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:25.528872 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:25.528873 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:25.528874 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:58:25.528927 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:25.528928 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:25.529782 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:25.529783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:25.531040 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:25.531043 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:25.531799 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:25.531800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:25.531892 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:25.531899 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 03:58:25.531963 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 03:58:25.532012 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":18,"thread_start_us":118,"threads_started":1}
I20260926 03:58:25.532528 1946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b522d5900f164785827e7d8a801e38e0" candidate_uuid: "113a195864e74a88a6b10daacba06759" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55485c830e8a45e68078486b3b60ec47" is_pre_election: true
I20260926 03:58:25.532727 1946 raft_consensus.cc:2454] T b522d5900f164785827e7d8a801e38e0 P 55485c830e8a45e68078486b3b60ec47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 113a195864e74a88a6b10daacba06759 in term 0.
I20260926 03:58:25.532748 1666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b522d5900f164785827e7d8a801e38e0" candidate_uuid: "113a195864e74a88a6b10daacba06759" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "266532c1f533453fb6df898015566b66" is_pre_election: true
I20260926 03:58:25.532959 1666 raft_consensus.cc:2454] T b522d5900f164785827e7d8a801e38e0 P 266532c1f533453fb6df898015566b66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 113a195864e74a88a6b10daacba06759 in term 0.
I20260926 03:58:25.532989 1738 leader_election.cc:302] T b522d5900f164785827e7d8a801e38e0 P 113a195864e74a88a6b10daacba06759 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 113a195864e74a88a6b10daacba06759, 55485c830e8a45e68078486b3b60ec47; no voters:
I20260926 03:58:25.533109 2021 raft_consensus.cc:2781] T b522d5900f164785827e7d8a801e38e0 P 113a195864e74a88a6b10daacba06759 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:58:25.533164 2021 raft_consensus.cc:482] T b522d5900f164785827e7d8a801e38e0 P 113a195864e74a88a6b10daacba06759 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:58:25.533192 2021 raft_consensus.cc:3037] T b522d5900f164785827e7d8a801e38e0 P 113a195864e74a88a6b10daacba06759 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:58:25.534126 2021 raft_consensus.cc:504] T b522d5900f164785827e7d8a801e38e0 P 113a195864e74a88a6b10daacba06759 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "113a195864e74a88a6b10daacba06759" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45311 } } peers { permanent_uuid: "55485c830e8a45e68078486b3b60ec47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38109 } } peers { permanent_uuid: "266532c1f533453fb6df898015566b66" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44309 } }
I20260926 03:58:25.534296 2021 leader_election.cc:288] T b522d5900f164785827e7d8a801e38e0 P 113a195864e74a88a6b10daacba06759 [CANDIDATE]: Term 1 election: Requested vote from peers 55485c830e8a45e68078486b3b60ec47 (127.0.98.3:38109), 266532c1f533453fb6df898015566b66 (127.0.98.1:44309)
I20260926 03:58:25.534435 1946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b522d5900f164785827e7d8a801e38e0" candidate_uuid: "113a195864e74a88a6b10daacba06759" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55485c830e8a45e68078486b3b60ec47"
I20260926 03:58:25.534461 1666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b522d5900f164785827e7d8a801e38e0" candidate_uuid: "113a195864e74a88a6b10daacba06759" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "266532c1f533453fb6df898015566b66"
I20260926 03:58:25.534510 1946 raft_consensus.cc:3037] T b522d5900f164785827e7d8a801e38e0 P 55485c830e8a45e68078486b3b60ec47 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:58:25.534541 1666 raft_consensus.cc:3037] T b522d5900f164785827e7d8a801e38e0 P 266532c1f533453fb6df898015566b66 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:58:25.535180 1946 raft_consensus.cc:2454] T b522d5900f164785827e7d8a801e38e0 P 55485c830e8a45e68078486b3b60ec47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 113a195864e74a88a6b10daacba06759 in term 1.
I20260926 03:58:25.535187 1666 raft_consensus.cc:2454] T b522d5900f164785827e7d8a801e38e0 P 266532c1f533453fb6df898015566b66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 113a195864e74a88a6b10daacba06759 in term 1.
I20260926 03:58:25.535360 1738 leader_election.cc:302] T b522d5900f164785827e7d8a801e38e0 P 113a195864e74a88a6b10daacba06759 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 113a195864e74a88a6b10daacba06759, 266532c1f533453fb6df898015566b66; no voters:
I20260926 03:58:25.535459 2021 raft_consensus.cc:2781] T b522d5900f164785827e7d8a801e38e0 P 113a195864e74a88a6b10daacba06759 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:58:25.535586 2021 raft_consensus.cc:686] T b522d5900f164785827e7d8a801e38e0 P 113a195864e74a88a6b10daacba06759 [term 1 LEADER]: Becoming Leader. State: Replica: 113a195864e74a88a6b10daacba06759, State: Running, Role: LEADER
I20260926 03:58:25.535676 2021 consensus_queue.cc:237] T b522d5900f164785827e7d8a801e38e0 P 113a195864e74a88a6b10daacba06759 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "113a195864e74a88a6b10daacba06759" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45311 } } peers { permanent_uuid: "55485c830e8a45e68078486b3b60ec47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38109 } } peers { permanent_uuid: "266532c1f533453fb6df898015566b66" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44309 } }
I20260926 03:58:25.536684 1509 catalog_manager.cc:5495] T b522d5900f164785827e7d8a801e38e0 P 113a195864e74a88a6b10daacba06759 reported cstate change: term changed from 0 to 1, leader changed from <none> to 113a195864e74a88a6b10daacba06759 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "113a195864e74a88a6b10daacba06759" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "113a195864e74a88a6b10daacba06759" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45311 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "55485c830e8a45e68078486b3b60ec47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "266532c1f533453fb6df898015566b66" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44309 } health_report { overall_health: UNKNOWN } } }
I20260926 03:58:25.551249 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:25.549116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52076 (local address 127.0.98.2:45311)
0926 03:58:25.549174 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:25.549176 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:25.549185 (+ 9us) server_negotiation.cc:407] Connection header received
0926 03:58:25.549243 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:25.549244 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:25.549260 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:25.549318 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:58:25.549452 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:25.549908 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:25.550624 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:25.550796 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:25.550856 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:25.551007 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:58:25.551091 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:58:25.551145 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:25.551186 (+ 41us) server_negotiation.cc:299] Negotiation successful
0926 03:58:25.551201 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 03:58:25.552845 1946 raft_consensus.cc:1261] T b522d5900f164785827e7d8a801e38e0 P 55485c830e8a45e68078486b3b60ec47 [term 1 FOLLOWER]: Refusing update from remote peer 113a195864e74a88a6b10daacba06759: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:58:25.552850 1666 raft_consensus.cc:1261] T b522d5900f164785827e7d8a801e38e0 P 266532c1f533453fb6df898015566b66 [term 1 FOLLOWER]: Refusing update from remote peer 113a195864e74a88a6b10daacba06759: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:58:25.553159 2021 consensus_queue.cc:1035] T b522d5900f164785827e7d8a801e38e0 P 113a195864e74a88a6b10daacba06759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "266532c1f533453fb6df898015566b66" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:58:25.553328 2025 consensus_queue.cc:1035] T b522d5900f164785827e7d8a801e38e0 P 113a195864e74a88a6b10daacba06759 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55485c830e8a45e68078486b3b60ec47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:58:25.555542 2028 mvcc.cc:205] Tried to move back new op lower bound from 7333458352341901312 to 7333458352275312640. Current Snapshot: MvccSnapshot[applied={T|T < 7333458352341901312}]
I20260926 03:58:25.556233 2030 mvcc.cc:205] Tried to move back new op lower bound from 7333458352341901312 to 7333458352275312640. Current Snapshot: MvccSnapshot[applied={T|T < 7333458352341901312}]
I20260926 03:58:25.556710 2029 mvcc.cc:205] Tried to move back new op lower bound from 7333458352341901312 to 7333458352275312640. Current Snapshot: MvccSnapshot[applied={T|T < 7333458352341901312 or (T in {7333458352341901312})}]
I20260926 03:58:25.559521 1508 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54472:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:58:25.559623 1508 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54472:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:58:25.560377 1508 catalog_manager.cc:5782] T 00000000000000000000000000000000 P df5fd5aad81a453790d435063bdd82d4: Sending DeleteTablet for 3 replicas of tablet b522d5900f164785827e7d8a801e38e0
I20260926 03:58:25.560695 1786 tablet_service.cc:1514] Processing DeleteTablet for tablet b522d5900f164785827e7d8a801e38e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:58:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52074
I20260926 03:58:25.560894 1926 tablet_service.cc:1514] Processing DeleteTablet for tablet b522d5900f164785827e7d8a801e38e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:58:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38124
I20260926 03:58:25.560945 1508 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54472:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:58:25.561092 1508 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:58:25.561178 1645 tablet_service.cc:1514] Processing DeleteTablet for tablet b522d5900f164785827e7d8a801e38e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:58:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58220
I20260926 03:58:25.561291 2038 tablet_replica.cc:330] T b522d5900f164785827e7d8a801e38e0 P 55485c830e8a45e68078486b3b60ec47: stopping tablet replica
I20260926 03:58:25.561378 2038 raft_consensus.cc:2229] T b522d5900f164785827e7d8a801e38e0 P 55485c830e8a45e68078486b3b60ec47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:58:25.561686 2037 tablet_replica.cc:330] T b522d5900f164785827e7d8a801e38e0 P 113a195864e74a88a6b10daacba06759: stopping tablet replica
I20260926 03:58:25.561782 2037 raft_consensus.cc:2229] T b522d5900f164785827e7d8a801e38e0 P 113a195864e74a88a6b10daacba06759 [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:58:25.561565 2039 tablet_replica.cc:330] T b522d5900f164785827e7d8a801e38e0 P 266532c1f533453fb6df898015566b66: stopping tablet replica
I20260926 03:58:25.562279 1786 tablet_service.cc:1467] Processing CreateTablet for tablet 01ed46de90f045bcb28819854c98a90d (DEFAULT_TABLE table=test-table [id=db13018cf24646989511433becbc4e0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:58:25.562325 1645 tablet_service.cc:1467] Processing CreateTablet for tablet 01ed46de90f045bcb28819854c98a90d (DEFAULT_TABLE table=test-table [id=db13018cf24646989511433becbc4e0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:58:25.562392 1786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01ed46de90f045bcb28819854c98a90d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:58:25.562420 1645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01ed46de90f045bcb28819854c98a90d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:58:25.563710 2017 tablet_bootstrap.cc:492] T 01ed46de90f045bcb28819854c98a90d P 113a195864e74a88a6b10daacba06759: Bootstrap starting.
I20260926 03:58:25.563737 2015 tablet_bootstrap.cc:492] T 01ed46de90f045bcb28819854c98a90d P 266532c1f533453fb6df898015566b66: Bootstrap starting.
I20260926 03:58:25.562119 2039 raft_consensus.cc:2229] T b522d5900f164785827e7d8a801e38e0 P 266532c1f533453fb6df898015566b66 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:58:25.563908 2039 raft_consensus.cc:2258] T b522d5900f164785827e7d8a801e38e0 P 266532c1f533453fb6df898015566b66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:58:25.564047 2037 raft_consensus.cc:2258] T b522d5900f164785827e7d8a801e38e0 P 113a195864e74a88a6b10daacba06759 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:58:25.564134 2015 tablet_bootstrap.cc:654] T 01ed46de90f045bcb28819854c98a90d P 266532c1f533453fb6df898015566b66: Neither blocks nor log segments found. Creating new log.
I20260926 03:58:25.564141 2017 tablet_bootstrap.cc:654] T 01ed46de90f045bcb28819854c98a90d P 113a195864e74a88a6b10daacba06759: Neither blocks nor log segments found. Creating new log.
I20260926 03:58:25.564507 2037 ts_tablet_manager.cc:1905] T b522d5900f164785827e7d8a801e38e0 P 113a195864e74a88a6b10daacba06759: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:58:25.564551 1926 tablet_service.cc:1467] Processing CreateTablet for tablet 01ed46de90f045bcb28819854c98a90d (DEFAULT_TABLE table=test-table [id=db13018cf24646989511433becbc4e0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:58:25.564671 1926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01ed46de90f045bcb28819854c98a90d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:58:25.564918 2038 raft_consensus.cc:2258] T b522d5900f164785827e7d8a801e38e0 P 55485c830e8a45e68078486b3b60ec47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:58:25.565405 2038 ts_tablet_manager.cc:1905] T b522d5900f164785827e7d8a801e38e0 P 55485c830e8a45e68078486b3b60ec47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:58:25.566113 2016 tablet_bootstrap.cc:492] T 01ed46de90f045bcb28819854c98a90d P 55485c830e8a45e68078486b3b60ec47: Bootstrap starting.
I20260926 03:58:25.566273 2039 ts_tablet_manager.cc:1905] T b522d5900f164785827e7d8a801e38e0 P 266532c1f533453fb6df898015566b66: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:58:25.566524 2016 tablet_bootstrap.cc:654] T 01ed46de90f045bcb28819854c98a90d P 55485c830e8a45e68078486b3b60ec47: Neither blocks nor log segments found. Creating new log.
I20260926 03:58:25.566534 2015 tablet_bootstrap.cc:492] T 01ed46de90f045bcb28819854c98a90d P 266532c1f533453fb6df898015566b66: No bootstrap required, opened a new log
I20260926 03:58:25.566573 2015 ts_tablet_manager.cc:1397] T 01ed46de90f045bcb28819854c98a90d P 266532c1f533453fb6df898015566b66: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 03:58:25.566711 2015 raft_consensus.cc:348] T 01ed46de90f045bcb28819854c98a90d P 266532c1f533453fb6df898015566b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "266532c1f533453fb6df898015566b66" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44309 } } peers { permanent_uuid: "113a195864e74a88a6b10daacba06759" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45311 } } peers { permanent_uuid: "55485c830e8a45e68078486b3b60ec47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38109 } }
I20260926 03:58:25.566769 2015 raft_consensus.cc:374] T 01ed46de90f045bcb28819854c98a90d P 266532c1f533453fb6df898015566b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:58:25.566789 2015 raft_consensus.cc:729] T 01ed46de90f045bcb28819854c98a90d P 266532c1f533453fb6df898015566b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 266532c1f533453fb6df898015566b66, State: Initialized, Role: FOLLOWER
I20260926 03:58:25.566831 2015 consensus_queue.cc:260] T 01ed46de90f045bcb28819854c98a90d P 266532c1f533453fb6df898015566b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "266532c1f533453fb6df898015566b66" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44309 } } peers { permanent_uuid: "113a195864e74a88a6b10daacba06759" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45311 } } peers { permanent_uuid: "55485c830e8a45e68078486b3b60ec47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38109 } }
I20260926 03:58:25.566938 2038 ts_tablet_manager.cc:1918] T b522d5900f164785827e7d8a801e38e0 P 55485c830e8a45e68078486b3b60ec47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:58:25.566998 2038 log.cc:1196] T b522d5900f164785827e7d8a801e38e0 P 55485c830e8a45e68078486b3b60ec47: Deleting WAL directory at /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/minicluster-data/ts-2/wal/wals/b522d5900f164785827e7d8a801e38e0
I20260926 03:58:25.567127 2015 ts_tablet_manager.cc:1428] T 01ed46de90f045bcb28819854c98a90d P 266532c1f533453fb6df898015566b66: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 03:58:25.567272 2038 ts_tablet_manager.cc:1939] T b522d5900f164785827e7d8a801e38e0 P 55485c830e8a45e68078486b3b60ec47: Deleting consensus metadata
I20260926 03:58:25.567468 2037 ts_tablet_manager.cc:1918] T b522d5900f164785827e7d8a801e38e0 P 113a195864e74a88a6b10daacba06759: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:58:25.567567 2037 log.cc:1196] T b522d5900f164785827e7d8a801e38e0 P 113a195864e74a88a6b10daacba06759: Deleting WAL directory at /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/minicluster-data/ts-1/wal/wals/b522d5900f164785827e7d8a801e38e0
I20260926 03:58:25.567799 2039 ts_tablet_manager.cc:1918] T b522d5900f164785827e7d8a801e38e0 P 266532c1f533453fb6df898015566b66: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:58:25.567822 2037 ts_tablet_manager.cc:1939] T b522d5900f164785827e7d8a801e38e0 P 113a195864e74a88a6b10daacba06759: Deleting consensus metadata
I20260926 03:58:25.567870 2039 log.cc:1196] T b522d5900f164785827e7d8a801e38e0 P 266532c1f533453fb6df898015566b66: Deleting WAL directory at /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/minicluster-data/ts-0/wal/wals/b522d5900f164785827e7d8a801e38e0
I20260926 03:58:25.567920 2016 tablet_bootstrap.cc:492] T 01ed46de90f045bcb28819854c98a90d P 55485c830e8a45e68078486b3b60ec47: No bootstrap required, opened a new log
I20260926 03:58:25.567971 2016 ts_tablet_manager.cc:1397] T 01ed46de90f045bcb28819854c98a90d P 55485c830e8a45e68078486b3b60ec47: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 03:58:25.568159 2039 ts_tablet_manager.cc:1939] T b522d5900f164785827e7d8a801e38e0 P 266532c1f533453fb6df898015566b66: Deleting consensus metadata
I20260926 03:58:25.568562 2017 tablet_bootstrap.cc:492] T 01ed46de90f045bcb28819854c98a90d P 113a195864e74a88a6b10daacba06759: No bootstrap required, opened a new log
I20260926 03:58:25.568539 2016 raft_consensus.cc:348] T 01ed46de90f045bcb28819854c98a90d P 55485c830e8a45e68078486b3b60ec47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "266532c1f533453fb6df898015566b66" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44309 } } peers { permanent_uuid: "113a195864e74a88a6b10daacba06759" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45311 } } peers { permanent_uuid: "55485c830e8a45e68078486b3b60ec47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38109 } }
I20260926 03:58:25.568610 2016 raft_consensus.cc:374] T 01ed46de90f045bcb28819854c98a90d P 55485c830e8a45e68078486b3b60ec47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:58:25.568614 2017 ts_tablet_manager.cc:1397] T 01ed46de90f045bcb28819854c98a90d P 113a195864e74a88a6b10daacba06759: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260926 03:58:25.568651 2016 raft_consensus.cc:729] T 01ed46de90f045bcb28819854c98a90d P 55485c830e8a45e68078486b3b60ec47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55485c830e8a45e68078486b3b60ec47, State: Initialized, Role: FOLLOWER
I20260926 03:58:25.568616 1495 catalog_manager.cc:4841] TS 113a195864e74a88a6b10daacba06759 (127.0.98.2:45311): tablet b522d5900f164785827e7d8a801e38e0 (table test-table [id=44e35391cd724a10a3672843c27c6d4c]) successfully deleted
I20260926 03:58:25.568698 2016 consensus_queue.cc:260] T 01ed46de90f045bcb28819854c98a90d P 55485c830e8a45e68078486b3b60ec47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "266532c1f533453fb6df898015566b66" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44309 } } peers { permanent_uuid: "113a195864e74a88a6b10daacba06759" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45311 } } peers { permanent_uuid: "55485c830e8a45e68078486b3b60ec47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38109 } }
I20260926 03:58:25.568782 2016 ts_tablet_manager.cc:1428] T 01ed46de90f045bcb28819854c98a90d P 55485c830e8a45e68078486b3b60ec47: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 03:58:25.568773 2017 raft_consensus.cc:348] T 01ed46de90f045bcb28819854c98a90d P 113a195864e74a88a6b10daacba06759 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "266532c1f533453fb6df898015566b66" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44309 } } peers { permanent_uuid: "113a195864e74a88a6b10daacba06759" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45311 } } peers { permanent_uuid: "55485c830e8a45e68078486b3b60ec47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38109 } }
I20260926 03:58:25.568835 2017 raft_consensus.cc:374] T 01ed46de90f045bcb28819854c98a90d P 113a195864e74a88a6b10daacba06759 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:58:25.568858 2017 raft_consensus.cc:729] T 01ed46de90f045bcb28819854c98a90d P 113a195864e74a88a6b10daacba06759 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 113a195864e74a88a6b10daacba06759, State: Initialized, Role: FOLLOWER
I20260926 03:58:25.568902 2017 consensus_queue.cc:260] T 01ed46de90f045bcb28819854c98a90d P 113a195864e74a88a6b10daacba06759 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "266532c1f533453fb6df898015566b66" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44309 } } peers { permanent_uuid: "113a195864e74a88a6b10daacba06759" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45311 } } peers { permanent_uuid: "55485c830e8a45e68078486b3b60ec47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38109 } }
I20260926 03:58:25.568987 2017 ts_tablet_manager.cc:1428] T 01ed46de90f045bcb28819854c98a90d P 113a195864e74a88a6b10daacba06759: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:25.569061 1494 catalog_manager.cc:4841] TS 55485c830e8a45e68078486b3b60ec47 (127.0.98.3:38109): tablet b522d5900f164785827e7d8a801e38e0 (table test-table [id=44e35391cd724a10a3672843c27c6d4c]) successfully deleted
I20260926 03:58:25.569131 1494 catalog_manager.cc:4841] TS 266532c1f533453fb6df898015566b66 (127.0.98.1:44309): tablet b522d5900f164785827e7d8a801e38e0 (table test-table [id=44e35391cd724a10a3672843c27c6d4c]) successfully deleted
W20260926 03:58:25.619940 1993 tablet.cc:2369] T 01ed46de90f045bcb28819854c98a90d P 55485c830e8a45e68078486b3b60ec47: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:58:25.768396 2023 raft_consensus.cc:482] T 01ed46de90f045bcb28819854c98a90d P 55485c830e8a45e68078486b3b60ec47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:58:25.768518 2023 raft_consensus.cc:504] T 01ed46de90f045bcb28819854c98a90d P 55485c830e8a45e68078486b3b60ec47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "266532c1f533453fb6df898015566b66" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44309 } } peers { permanent_uuid: "113a195864e74a88a6b10daacba06759" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45311 } } peers { permanent_uuid: "55485c830e8a45e68078486b3b60ec47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38109 } }
I20260926 03:58:25.768903 2023 leader_election.cc:288] T 01ed46de90f045bcb28819854c98a90d P 55485c830e8a45e68078486b3b60ec47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 266532c1f533453fb6df898015566b66 (127.0.98.1:44309), 113a195864e74a88a6b10daacba06759 (127.0.98.2:45311)
I20260926 03:58:25.773136 1885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:25.769096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45311 (local address 127.0.98.3:32863)
0926 03:58:25.769168 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:25.769182 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:25.769233 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:25.769557 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:25.769558 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:25.769559 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:58:25.769634 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:25.769636 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:25.770546 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:25.770547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:25.771904 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:25.771907 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:25.772806 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:25.772807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:25.772902 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:25.772909 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 03:58:25.772977 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 03:58:25.773021 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":109}
I20260926 03:58:25.773157 2043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:25.769120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44309 (local address 127.0.98.3:46407)
0926 03:58:25.769262 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:25.769281 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:25.769338 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:25.769557 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:25.769558 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:25.769559 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:58:25.769634 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:25.769636 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:25.770546 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:25.770547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:25.771906 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:25.771908 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:25.772829 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:25.772830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:25.772902 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:25.772909 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 03:58:25.772977 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 03:58:25.773019 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":39,"thread_start_us":73,"threads_started":1}
I20260926 03:58:25.773213 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:25.769155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:32863 (local address 127.0.98.2:45311)
0926 03:58:25.769205 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:25.769208 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:25.769223 (+ 15us) server_negotiation.cc:407] Connection header received
0926 03:58:25.769311 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:25.769311 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:25.769339 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:25.769414 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:58:25.769711 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:25.770459 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:25.771997 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:25.772607 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:25.772718 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:25.772728 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:25.773107 (+ 379us) server_negotiation.cc:299] Negotiation successful
0926 03:58:25.773146 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 03:58:25.773213 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:25.769187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46407 (local address 127.0.98.1:44309)
0926 03:58:25.769232 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:25.769234 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:25.769343 (+ 109us) server_negotiation.cc:407] Connection header received
0926 03:58:25.769377 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:25.769378 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:25.769388 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:25.769443 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:58:25.769710 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:25.770452 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:25.771996 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:25.772591 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:25.772712 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:25.772724 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:25.773107 (+ 383us) server_negotiation.cc:299] Negotiation successful
0926 03:58:25.773151 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 03:58:25.773432 1806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ed46de90f045bcb28819854c98a90d" candidate_uuid: "55485c830e8a45e68078486b3b60ec47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "113a195864e74a88a6b10daacba06759" is_pre_election: true
I20260926 03:58:25.773532 1806 raft_consensus.cc:2454] T 01ed46de90f045bcb28819854c98a90d P 113a195864e74a88a6b10daacba06759 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55485c830e8a45e68078486b3b60ec47 in term 0.
I20260926 03:58:25.773732 1879 leader_election.cc:302] T 01ed46de90f045bcb28819854c98a90d P 55485c830e8a45e68078486b3b60ec47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 113a195864e74a88a6b10daacba06759, 55485c830e8a45e68078486b3b60ec47; no voters:
I20260926 03:58:25.773862 2023 raft_consensus.cc:2781] T 01ed46de90f045bcb28819854c98a90d P 55485c830e8a45e68078486b3b60ec47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:58:25.773908 2023 raft_consensus.cc:482] T 01ed46de90f045bcb28819854c98a90d P 55485c830e8a45e68078486b3b60ec47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:58:25.773936 2023 raft_consensus.cc:3037] T 01ed46de90f045bcb28819854c98a90d P 55485c830e8a45e68078486b3b60ec47 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:58:25.773952 1666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ed46de90f045bcb28819854c98a90d" candidate_uuid: "55485c830e8a45e68078486b3b60ec47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "266532c1f533453fb6df898015566b66" is_pre_election: true
I20260926 03:58:25.774039 1666 raft_consensus.cc:2454] T 01ed46de90f045bcb28819854c98a90d P 266532c1f533453fb6df898015566b66 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 55485c830e8a45e68078486b3b60ec47 in term 0.
I20260926 03:58:25.774653 2023 raft_consensus.cc:504] T 01ed46de90f045bcb28819854c98a90d P 55485c830e8a45e68078486b3b60ec47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "266532c1f533453fb6df898015566b66" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44309 } } peers { permanent_uuid: "113a195864e74a88a6b10daacba06759" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45311 } } peers { permanent_uuid: "55485c830e8a45e68078486b3b60ec47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38109 } }
I20260926 03:58:25.774794 2023 leader_election.cc:288] T 01ed46de90f045bcb28819854c98a90d P 55485c830e8a45e68078486b3b60ec47 [CANDIDATE]: Term 1 election: Requested vote from peers 266532c1f533453fb6df898015566b66 (127.0.98.1:44309), 113a195864e74a88a6b10daacba06759 (127.0.98.2:45311)
I20260926 03:58:25.774947 1666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ed46de90f045bcb28819854c98a90d" candidate_uuid: "55485c830e8a45e68078486b3b60ec47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "266532c1f533453fb6df898015566b66"
I20260926 03:58:25.775020 1666 raft_consensus.cc:3037] T 01ed46de90f045bcb28819854c98a90d P 266532c1f533453fb6df898015566b66 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:58:25.774991 1806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01ed46de90f045bcb28819854c98a90d" candidate_uuid: "55485c830e8a45e68078486b3b60ec47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "113a195864e74a88a6b10daacba06759"
I20260926 03:58:25.775107 1806 raft_consensus.cc:3037] T 01ed46de90f045bcb28819854c98a90d P 113a195864e74a88a6b10daacba06759 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:58:25.775620 1666 raft_consensus.cc:2454] T 01ed46de90f045bcb28819854c98a90d P 266532c1f533453fb6df898015566b66 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55485c830e8a45e68078486b3b60ec47 in term 1.
I20260926 03:58:25.775794 1878 leader_election.cc:302] T 01ed46de90f045bcb28819854c98a90d P 55485c830e8a45e68078486b3b60ec47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 266532c1f533453fb6df898015566b66, 55485c830e8a45e68078486b3b60ec47; no voters:
I20260926 03:58:25.775837 1806 raft_consensus.cc:2454] T 01ed46de90f045bcb28819854c98a90d P 113a195864e74a88a6b10daacba06759 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55485c830e8a45e68078486b3b60ec47 in term 1.
I20260926 03:58:25.775893 2023 raft_consensus.cc:2781] T 01ed46de90f045bcb28819854c98a90d P 55485c830e8a45e68078486b3b60ec47 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:58:25.776023 2023 raft_consensus.cc:686] T 01ed46de90f045bcb28819854c98a90d P 55485c830e8a45e68078486b3b60ec47 [term 1 LEADER]: Becoming Leader. State: Replica: 55485c830e8a45e68078486b3b60ec47, State: Running, Role: LEADER
I20260926 03:58:25.776098 2023 consensus_queue.cc:237] T 01ed46de90f045bcb28819854c98a90d P 55485c830e8a45e68078486b3b60ec47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "266532c1f533453fb6df898015566b66" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44309 } } peers { permanent_uuid: "113a195864e74a88a6b10daacba06759" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45311 } } peers { permanent_uuid: "55485c830e8a45e68078486b3b60ec47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38109 } }
I20260926 03:58:25.776839 1509 catalog_manager.cc:5495] T 01ed46de90f045bcb28819854c98a90d P 55485c830e8a45e68078486b3b60ec47 reported cstate change: term changed from 0 to 1, leader changed from <none> to 55485c830e8a45e68078486b3b60ec47 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "55485c830e8a45e68078486b3b60ec47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "266532c1f533453fb6df898015566b66" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "113a195864e74a88a6b10daacba06759" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "55485c830e8a45e68078486b3b60ec47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38109 } health_report { overall_health: HEALTHY } } }
Sep 26 03:58:25 dist-test-slave-nhpw krb5kdc[1464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395104, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 03:58:25.816267 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:25.809320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54482 (local address 127.0.98.62:46765)
0926 03:58:25.809411 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:25.809413 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:25.809436 (+ 23us) server_negotiation.cc:407] Connection header received
0926 03:58:25.809509 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:25.809509 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:25.809518 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:25.809569 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:25.810012 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:25.810505 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:25.811226 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:25.811416 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:25.811521 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:25.813629 (+ 2108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:25.813631 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:25.813631 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:25.813637 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:25.815076 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:25.815527 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:25.815528 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:25.815528 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:25.815575 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:25.815804 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:25.815805 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:25.815805 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:25.815911 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:25.815966 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:25.816144 (+ 178us) server_negotiation.cc:299] Negotiation successful
0926 03:58:25.816210 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 03:58:25.816401 2051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:25.809221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46765 (local address 127.0.0.1:54482)
0926 03:58:25.809382 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:25.809397 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:25.809451 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:25.809587 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:25.809588 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:25.809895 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:25.809944 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:25.809946 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:25.810578 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:25.810579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:25.811121 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:25.811123 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:25.811545 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:25.811546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:25.811592 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:25.812173 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:25.812178 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:25.813539 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:58:25.815176 (+ 1637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:25.815182 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:25.815184 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:25.815450 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:25.815641 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:25.815642 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:25.815642 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:25.815715 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:25.815980 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:58:25.815986 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:58:25.816041 (+ 55us) client_negotiation.cc:769] Sending connection context
0926 03:58:25.816119 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 03:58:25.816293 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":67,"thread_start_us":81,"threads_started":1}
I20260926 03:58:25.818756 1509 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54482:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 03:58:25.818938 1509 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:58:25.820371 1926 tablet_service.cc:1467] Processing CreateTablet for tablet cc75a846028445b081beab29dd13adc4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8e912b9cc42d4382bf85ebb256927669]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 03:58:25.820392 1786 tablet_service.cc:1467] Processing CreateTablet for tablet cc75a846028445b081beab29dd13adc4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8e912b9cc42d4382bf85ebb256927669]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 03:58:25.820429 1645 tablet_service.cc:1467] Processing CreateTablet for tablet cc75a846028445b081beab29dd13adc4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8e912b9cc42d4382bf85ebb256927669]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 03:58:25.820534 1926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc75a846028445b081beab29dd13adc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:58:25.820560 1645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc75a846028445b081beab29dd13adc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:58:25.820564 1786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc75a846028445b081beab29dd13adc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:58:25.821709 2017 tablet_bootstrap.cc:492] T cc75a846028445b081beab29dd13adc4 P 113a195864e74a88a6b10daacba06759: Bootstrap starting.
I20260926 03:58:25.822005 2016 tablet_bootstrap.cc:492] T cc75a846028445b081beab29dd13adc4 P 55485c830e8a45e68078486b3b60ec47: Bootstrap starting.
I20260926 03:58:25.822090 2017 tablet_bootstrap.cc:654] T cc75a846028445b081beab29dd13adc4 P 113a195864e74a88a6b10daacba06759: Neither blocks nor log segments found. Creating new log.
I20260926 03:58:25.822410 2016 tablet_bootstrap.cc:654] T cc75a846028445b081beab29dd13adc4 P 55485c830e8a45e68078486b3b60ec47: Neither blocks nor log segments found. Creating new log.
I20260926 03:58:25.822644 2017 tablet_bootstrap.cc:492] T cc75a846028445b081beab29dd13adc4 P 113a195864e74a88a6b10daacba06759: No bootstrap required, opened a new log
I20260926 03:58:25.822703 2017 ts_tablet_manager.cc:1397] T cc75a846028445b081beab29dd13adc4 P 113a195864e74a88a6b10daacba06759: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 03:58:25.822785 2015 tablet_bootstrap.cc:492] T cc75a846028445b081beab29dd13adc4 P 266532c1f533453fb6df898015566b66: Bootstrap starting.
I20260926 03:58:25.822813 2017 raft_consensus.cc:348] T cc75a846028445b081beab29dd13adc4 P 113a195864e74a88a6b10daacba06759 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "266532c1f533453fb6df898015566b66" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44309 } } peers { permanent_uuid: "113a195864e74a88a6b10daacba06759" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45311 } } peers { permanent_uuid: "55485c830e8a45e68078486b3b60ec47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38109 } }
I20260926 03:58:25.822882 2017 raft_consensus.cc:374] T cc75a846028445b081beab29dd13adc4 P 113a195864e74a88a6b10daacba06759 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:58:25.822908 2017 raft_consensus.cc:729] T cc75a846028445b081beab29dd13adc4 P 113a195864e74a88a6b10daacba06759 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 113a195864e74a88a6b10daacba06759, State: Initialized, Role: FOLLOWER
I20260926 03:58:25.823390 2017 consensus_queue.cc:260] T cc75a846028445b081beab29dd13adc4 P 113a195864e74a88a6b10daacba06759 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "266532c1f533453fb6df898015566b66" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44309 } } peers { permanent_uuid: "113a195864e74a88a6b10daacba06759" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45311 } } peers { permanent_uuid: "55485c830e8a45e68078486b3b60ec47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38109 } }
I20260926 03:58:25.823523 2017 ts_tablet_manager.cc:1428] T cc75a846028445b081beab29dd13adc4 P 113a195864e74a88a6b10daacba06759: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 03:58:25.823570 2025 tablet_replica.cc:439] T cc75a846028445b081beab29dd13adc4 P 113a195864e74a88a6b10daacba06759: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "266532c1f533453fb6df898015566b66" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44309 } } peers { permanent_uuid: "113a195864e74a88a6b10daacba06759" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45311 } } peers { permanent_uuid: "55485c830e8a45e68078486b3b60ec47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38109 } } }
I20260926 03:58:25.823661 2025 tablet_replica.cc:442] T cc75a846028445b081beab29dd13adc4 P 113a195864e74a88a6b10daacba06759: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:58:25.823714 2015 tablet_bootstrap.cc:654] T cc75a846028445b081beab29dd13adc4 P 266532c1f533453fb6df898015566b66: Neither blocks nor log segments found. Creating new log.
I20260926 03:58:25.824215 2016 tablet_bootstrap.cc:492] T cc75a846028445b081beab29dd13adc4 P 55485c830e8a45e68078486b3b60ec47: No bootstrap required, opened a new log
I20260926 03:58:25.824273 2016 ts_tablet_manager.cc:1397] T cc75a846028445b081beab29dd13adc4 P 55485c830e8a45e68078486b3b60ec47: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 03:58:25.824584 2016 raft_consensus.cc:348] T cc75a846028445b081beab29dd13adc4 P 55485c830e8a45e68078486b3b60ec47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "266532c1f533453fb6df898015566b66" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44309 } } peers { permanent_uuid: "113a195864e74a88a6b10daacba06759" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45311 } } peers { permanent_uuid: "55485c830e8a45e68078486b3b60ec47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38109 } }
I20260926 03:58:25.824707 2016 raft_consensus.cc:374] T cc75a846028445b081beab29dd13adc4 P 55485c830e8a45e68078486b3b60ec47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:58:25.824728 2016 raft_consensus.cc:729] T cc75a846028445b081beab29dd13adc4 P 55485c830e8a45e68078486b3b60ec47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55485c830e8a45e68078486b3b60ec47, State: Initialized, Role: FOLLOWER
I20260926 03:58:25.824795 2016 consensus_queue.cc:260] T cc75a846028445b081beab29dd13adc4 P 55485c830e8a45e68078486b3b60ec47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "266532c1f533453fb6df898015566b66" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44309 } } peers { permanent_uuid: "113a195864e74a88a6b10daacba06759" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45311 } } peers { permanent_uuid: "55485c830e8a45e68078486b3b60ec47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38109 } }
I20260926 03:58:25.824887 2016 ts_tablet_manager.cc:1428] T cc75a846028445b081beab29dd13adc4 P 55485c830e8a45e68078486b3b60ec47: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 03:58:25.824970 2044 tablet_replica.cc:439] T cc75a846028445b081beab29dd13adc4 P 55485c830e8a45e68078486b3b60ec47: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "266532c1f533453fb6df898015566b66" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44309 } } peers { permanent_uuid: "113a195864e74a88a6b10daacba06759" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45311 } } peers { permanent_uuid: "55485c830e8a45e68078486b3b60ec47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38109 } } }
I20260926 03:58:25.825067 2044 tablet_replica.cc:442] T cc75a846028445b081beab29dd13adc4 P 55485c830e8a45e68078486b3b60ec47: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:58:25.825116 2015 tablet_bootstrap.cc:492] T cc75a846028445b081beab29dd13adc4 P 266532c1f533453fb6df898015566b66: No bootstrap required, opened a new log
I20260926 03:58:25.825165 2015 ts_tablet_manager.cc:1397] T cc75a846028445b081beab29dd13adc4 P 266532c1f533453fb6df898015566b66: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 03:58:25.825296 2015 raft_consensus.cc:348] T cc75a846028445b081beab29dd13adc4 P 266532c1f533453fb6df898015566b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "266532c1f533453fb6df898015566b66" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44309 } } peers { permanent_uuid: "113a195864e74a88a6b10daacba06759" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45311 } } peers { permanent_uuid: "55485c830e8a45e68078486b3b60ec47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38109 } }
I20260926 03:58:25.825371 2015 raft_consensus.cc:374] T cc75a846028445b081beab29dd13adc4 P 266532c1f533453fb6df898015566b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:58:25.825392 2015 raft_consensus.cc:729] T cc75a846028445b081beab29dd13adc4 P 266532c1f533453fb6df898015566b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 266532c1f533453fb6df898015566b66, State: Initialized, Role: FOLLOWER
I20260926 03:58:25.825443 2015 consensus_queue.cc:260] T cc75a846028445b081beab29dd13adc4 P 266532c1f533453fb6df898015566b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "266532c1f533453fb6df898015566b66" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44309 } } peers { permanent_uuid: "113a195864e74a88a6b10daacba06759" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45311 } } peers { permanent_uuid: "55485c830e8a45e68078486b3b60ec47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38109 } }
I20260926 03:58:25.825541 2015 ts_tablet_manager.cc:1428] T cc75a846028445b081beab29dd13adc4 P 266532c1f533453fb6df898015566b66: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 03:58:25.825693 2020 tablet_replica.cc:439] T cc75a846028445b081beab29dd13adc4 P 266532c1f533453fb6df898015566b66: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "266532c1f533453fb6df898015566b66" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44309 } } peers { permanent_uuid: "113a195864e74a88a6b10daacba06759" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45311 } } peers { permanent_uuid: "55485c830e8a45e68078486b3b60ec47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38109 } } }
I20260926 03:58:25.825793 2020 tablet_replica.cc:442] T cc75a846028445b081beab29dd13adc4 P 266532c1f533453fb6df898015566b66: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:58:26.011365 2020 raft_consensus.cc:482] T cc75a846028445b081beab29dd13adc4 P 266532c1f533453fb6df898015566b66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:58:26.011485 2020 raft_consensus.cc:504] T cc75a846028445b081beab29dd13adc4 P 266532c1f533453fb6df898015566b66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "266532c1f533453fb6df898015566b66" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44309 } } peers { permanent_uuid: "113a195864e74a88a6b10daacba06759" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45311 } } peers { permanent_uuid: "55485c830e8a45e68078486b3b60ec47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38109 } }
I20260926 03:58:26.011915 2020 leader_election.cc:288] T cc75a846028445b081beab29dd13adc4 P 266532c1f533453fb6df898015566b66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 113a195864e74a88a6b10daacba06759 (127.0.98.2:45311), 55485c830e8a45e68078486b3b60ec47 (127.0.98.3:38109)
I20260926 03:58:26.014993 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:26.012384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48297 (local address 127.0.98.3:38109)
0926 03:58:26.012431 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:26.012433 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:26.012441 (+ 8us) server_negotiation.cc:407] Connection header received
0926 03:58:26.012472 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:26.012472 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:26.012479 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:26.012546 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:58:26.012737 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:26.013211 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:26.014268 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:26.014620 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:26.014703 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:26.014712 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:26.014879 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 03:58:26.014914 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 03:58:26.014992 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:26.012151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56757 (local address 127.0.98.2:45311)
0926 03:58:26.012224 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:26.012226 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:26.012360 (+ 134us) server_negotiation.cc:407] Connection header received
0926 03:58:26.012449 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:26.012450 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:26.012466 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:26.012546 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:58:26.012733 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:26.013211 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:26.014277 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:26.014637 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:26.014714 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:26.014724 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:26.014881 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 03:58:26.014916 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 03:58:26.015054 2056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:26.012150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38109 (local address 127.0.98.1:48297)
0926 03:58:26.012294 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:26.012303 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:26.012378 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:26.012550 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:26.012551 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:26.012553 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:58:26.012677 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:26.012679 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:26.013308 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:26.013309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:26.014170 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:26.014172 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:26.014713 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:26.014714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:26.014799 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:26.014803 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 03:58:26.014874 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 03:58:26.014918 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":32,"thread_start_us":95,"threads_started":1}
I20260926 03:58:26.015244 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:26.012105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45311 (local address 127.0.98.1:56757)
0926 03:58:26.012288 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:26.012301 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:26.012368 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:26.012548 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:26.012548 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:26.012550 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:58:26.012659 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:26.012661 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:26.013303 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:26.013303 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:26.014169 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:26.014171 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:26.014728 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:26.014729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:26.014807 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:26.014810 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 03:58:26.014876 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 03:58:26.014920 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":61,"thread_start_us":98,"threads_started":1}
I20260926 03:58:26.015362 1946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc75a846028445b081beab29dd13adc4" candidate_uuid: "266532c1f533453fb6df898015566b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55485c830e8a45e68078486b3b60ec47" is_pre_election: true
I20260926 03:58:26.015465 1946 raft_consensus.cc:2454] T cc75a846028445b081beab29dd13adc4 P 55485c830e8a45e68078486b3b60ec47 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 266532c1f533453fb6df898015566b66 in term 0.
I20260926 03:58:26.015522 1806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc75a846028445b081beab29dd13adc4" candidate_uuid: "266532c1f533453fb6df898015566b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "113a195864e74a88a6b10daacba06759" is_pre_election: true
I20260926 03:58:26.015609 1806 raft_consensus.cc:2454] T cc75a846028445b081beab29dd13adc4 P 113a195864e74a88a6b10daacba06759 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 266532c1f533453fb6df898015566b66 in term 0.
I20260926 03:58:26.015663 1598 leader_election.cc:302] T cc75a846028445b081beab29dd13adc4 P 266532c1f533453fb6df898015566b66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 266532c1f533453fb6df898015566b66, 55485c830e8a45e68078486b3b60ec47; no voters:
I20260926 03:58:26.015805 2020 raft_consensus.cc:2781] T cc75a846028445b081beab29dd13adc4 P 266532c1f533453fb6df898015566b66 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:58:26.015858 2020 raft_consensus.cc:482] T cc75a846028445b081beab29dd13adc4 P 266532c1f533453fb6df898015566b66 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:58:26.015890 2020 raft_consensus.cc:3037] T cc75a846028445b081beab29dd13adc4 P 266532c1f533453fb6df898015566b66 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:58:26.016566 2020 raft_consensus.cc:504] T cc75a846028445b081beab29dd13adc4 P 266532c1f533453fb6df898015566b66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "266532c1f533453fb6df898015566b66" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44309 } } peers { permanent_uuid: "113a195864e74a88a6b10daacba06759" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45311 } } peers { permanent_uuid: "55485c830e8a45e68078486b3b60ec47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38109 } }
I20260926 03:58:26.016728 2020 leader_election.cc:288] T cc75a846028445b081beab29dd13adc4 P 266532c1f533453fb6df898015566b66 [CANDIDATE]: Term 1 election: Requested vote from peers 113a195864e74a88a6b10daacba06759 (127.0.98.2:45311), 55485c830e8a45e68078486b3b60ec47 (127.0.98.3:38109)
I20260926 03:58:26.016875 1806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc75a846028445b081beab29dd13adc4" candidate_uuid: "266532c1f533453fb6df898015566b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "113a195864e74a88a6b10daacba06759"
I20260926 03:58:26.016943 1946 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc75a846028445b081beab29dd13adc4" candidate_uuid: "266532c1f533453fb6df898015566b66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55485c830e8a45e68078486b3b60ec47"
I20260926 03:58:26.016990 1806 raft_consensus.cc:3037] T cc75a846028445b081beab29dd13adc4 P 113a195864e74a88a6b10daacba06759 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:58:26.017028 1946 raft_consensus.cc:3037] T cc75a846028445b081beab29dd13adc4 P 55485c830e8a45e68078486b3b60ec47 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:58:26.017572 1806 raft_consensus.cc:2454] T cc75a846028445b081beab29dd13adc4 P 113a195864e74a88a6b10daacba06759 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 266532c1f533453fb6df898015566b66 in term 1.
I20260926 03:58:26.017621 1946 raft_consensus.cc:2454] T cc75a846028445b081beab29dd13adc4 P 55485c830e8a45e68078486b3b60ec47 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 266532c1f533453fb6df898015566b66 in term 1.
I20260926 03:58:26.017773 1599 leader_election.cc:302] T cc75a846028445b081beab29dd13adc4 P 266532c1f533453fb6df898015566b66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 113a195864e74a88a6b10daacba06759, 266532c1f533453fb6df898015566b66; no voters:
I20260926 03:58:26.017923 2020 raft_consensus.cc:2781] T cc75a846028445b081beab29dd13adc4 P 266532c1f533453fb6df898015566b66 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:58:26.018064 2020 raft_consensus.cc:686] T cc75a846028445b081beab29dd13adc4 P 266532c1f533453fb6df898015566b66 [term 1 LEADER]: Becoming Leader. State: Replica: 266532c1f533453fb6df898015566b66, State: Running, Role: LEADER
I20260926 03:58:26.018118 2020 consensus_queue.cc:237] T cc75a846028445b081beab29dd13adc4 P 266532c1f533453fb6df898015566b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "266532c1f533453fb6df898015566b66" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44309 } } peers { permanent_uuid: "113a195864e74a88a6b10daacba06759" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45311 } } peers { permanent_uuid: "55485c830e8a45e68078486b3b60ec47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38109 } }
I20260926 03:58:26.018491 2057 tablet_replica.cc:439] T cc75a846028445b081beab29dd13adc4 P 266532c1f533453fb6df898015566b66: TxnStatusTablet state changed. Reason: New leader 266532c1f533453fb6df898015566b66. Latest consensus state: current_term: 1 leader_uuid: "266532c1f533453fb6df898015566b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "266532c1f533453fb6df898015566b66" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44309 } } peers { permanent_uuid: "113a195864e74a88a6b10daacba06759" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45311 } } peers { permanent_uuid: "55485c830e8a45e68078486b3b60ec47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38109 } } }
I20260926 03:58:26.018597 2057 tablet_replica.cc:442] T cc75a846028445b081beab29dd13adc4 P 266532c1f533453fb6df898015566b66: This TxnStatusTablet replica's current role is: LEADER
I20260926 03:58:26.018782 2059 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 03:58:26.018896 2059 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 03:58:26.018875 1509 catalog_manager.cc:5495] T cc75a846028445b081beab29dd13adc4 P 266532c1f533453fb6df898015566b66 reported cstate change: term changed from 0 to 1, leader changed from <none> to 266532c1f533453fb6df898015566b66 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "266532c1f533453fb6df898015566b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "266532c1f533453fb6df898015566b66" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44309 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "113a195864e74a88a6b10daacba06759" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45311 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "55485c830e8a45e68078486b3b60ec47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38109 } health_report { overall_health: UNKNOWN } } }
I20260926 03:58:26.051942 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:26.049373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58234 (local address 127.0.98.1:44309)
0926 03:58:26.049422 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:26.049424 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:26.049442 (+ 18us) server_negotiation.cc:407] Connection header received
0926 03:58:26.049517 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:26.049518 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:26.049538 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:26.049602 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:58:26.049774 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:26.050258 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:26.051016 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:26.051220 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:26.051306 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:26.051500 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:58:26.051599 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:58:26.051790 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:26.051845 (+ 55us) server_negotiation.cc:299] Negotiation successful
0926 03:58:26.051867 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 03:58:26.051970 2051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:26.049286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44309 (local address 127.0.0.1:58234)
0926 03:58:26.049366 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:26.049379 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:26.049431 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:26.049614 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:26.049615 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:26.049616 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 03:58:26.049683 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:26.049686 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:26.050341 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:26.050343 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:26.050920 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:26.050921 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:26.051306 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:26.051307 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:26.051373 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:26.051390 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 03:58:26.051768 (+ 378us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 03:58:26.051778 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 03:58:26.051845 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 03:58:26.051896 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 03:58:26.053210 1946 raft_consensus.cc:1261] T cc75a846028445b081beab29dd13adc4 P 55485c830e8a45e68078486b3b60ec47 [term 1 FOLLOWER]: Refusing update from remote peer 266532c1f533453fb6df898015566b66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:58:26.053308 1806 raft_consensus.cc:1261] T cc75a846028445b081beab29dd13adc4 P 113a195864e74a88a6b10daacba06759 [term 1 FOLLOWER]: Refusing update from remote peer 266532c1f533453fb6df898015566b66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:58:26.053464 2057 consensus_queue.cc:1035] T cc75a846028445b081beab29dd13adc4 P 266532c1f533453fb6df898015566b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "55485c830e8a45e68078486b3b60ec47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:58:26.053565 2020 consensus_queue.cc:1035] T cc75a846028445b081beab29dd13adc4 P 266532c1f533453fb6df898015566b66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "113a195864e74a88a6b10daacba06759" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:58:26.054056 2044 tablet_replica.cc:439] T cc75a846028445b081beab29dd13adc4 P 55485c830e8a45e68078486b3b60ec47: TxnStatusTablet state changed. Reason: New leader 266532c1f533453fb6df898015566b66. Latest consensus state: current_term: 1 leader_uuid: "266532c1f533453fb6df898015566b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "266532c1f533453fb6df898015566b66" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44309 } } peers { permanent_uuid: "113a195864e74a88a6b10daacba06759" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45311 } } peers { permanent_uuid: "55485c830e8a45e68078486b3b60ec47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38109 } } }
I20260926 03:58:26.054147 2044 tablet_replica.cc:442] T cc75a846028445b081beab29dd13adc4 P 55485c830e8a45e68078486b3b60ec47: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:58:26.054450 2025 tablet_replica.cc:439] T cc75a846028445b081beab29dd13adc4 P 113a195864e74a88a6b10daacba06759: TxnStatusTablet state changed. Reason: New leader 266532c1f533453fb6df898015566b66. Latest consensus state: current_term: 1 leader_uuid: "266532c1f533453fb6df898015566b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "266532c1f533453fb6df898015566b66" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44309 } } peers { permanent_uuid: "113a195864e74a88a6b10daacba06759" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45311 } } peers { permanent_uuid: "55485c830e8a45e68078486b3b60ec47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38109 } } }
I20260926 03:58:26.054550 2025 tablet_replica.cc:442] T cc75a846028445b081beab29dd13adc4 P 113a195864e74a88a6b10daacba06759: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:58:26.055152 2057 tablet_replica.cc:439] T cc75a846028445b081beab29dd13adc4 P 266532c1f533453fb6df898015566b66: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "266532c1f533453fb6df898015566b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "266532c1f533453fb6df898015566b66" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44309 } } peers { permanent_uuid: "113a195864e74a88a6b10daacba06759" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45311 } } peers { permanent_uuid: "55485c830e8a45e68078486b3b60ec47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38109 } } }
I20260926 03:58:26.055238 2057 tablet_replica.cc:442] T cc75a846028445b081beab29dd13adc4 P 266532c1f533453fb6df898015566b66: This TxnStatusTablet replica's current role is: LEADER
I20260926 03:58:26.055294 2057 tablet_replica.cc:439] T cc75a846028445b081beab29dd13adc4 P 266532c1f533453fb6df898015566b66: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "266532c1f533453fb6df898015566b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "266532c1f533453fb6df898015566b66" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44309 } } peers { permanent_uuid: "113a195864e74a88a6b10daacba06759" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45311 } } peers { permanent_uuid: "55485c830e8a45e68078486b3b60ec47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38109 } } }
I20260926 03:58:26.055351 2057 tablet_replica.cc:442] T cc75a846028445b081beab29dd13adc4 P 266532c1f533453fb6df898015566b66: This TxnStatusTablet replica's current role is: LEADER
I20260926 03:58:26.055581 2025 tablet_replica.cc:439] T cc75a846028445b081beab29dd13adc4 P 113a195864e74a88a6b10daacba06759: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "266532c1f533453fb6df898015566b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "266532c1f533453fb6df898015566b66" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44309 } } peers { permanent_uuid: "113a195864e74a88a6b10daacba06759" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45311 } } peers { permanent_uuid: "55485c830e8a45e68078486b3b60ec47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38109 } } }
I20260926 03:58:26.055656 2025 tablet_replica.cc:442] T cc75a846028445b081beab29dd13adc4 P 113a195864e74a88a6b10daacba06759: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:58:26.056491 2044 tablet_replica.cc:439] T cc75a846028445b081beab29dd13adc4 P 55485c830e8a45e68078486b3b60ec47: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "266532c1f533453fb6df898015566b66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "266532c1f533453fb6df898015566b66" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44309 } } peers { permanent_uuid: "113a195864e74a88a6b10daacba06759" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45311 } } peers { permanent_uuid: "55485c830e8a45e68078486b3b60ec47" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38109 } } }
I20260926 03:58:26.056593 2044 tablet_replica.cc:442] T cc75a846028445b081beab29dd13adc4 P 55485c830e8a45e68078486b3b60ec47: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:58:26.059510 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:26.057052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38134 (local address 127.0.98.3:38109)
0926 03:58:26.057094 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:26.057096 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:26.057224 (+ 128us) server_negotiation.cc:407] Connection header received
0926 03:58:26.057314 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:26.057315 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:26.057333 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:26.057419 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:58:26.057516 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:26.057977 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:26.058724 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:26.058935 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:26.059019 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:26.059177 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:58:26.059280 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:58:26.059355 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:26.059406 (+ 51us) server_negotiation.cc:299] Negotiation successful
0926 03:58:26.059424 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 03:58:26.064347 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:26.061856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44999 (local address 127.0.98.1:44309)
0926 03:58:26.061912 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:26.061914 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:26.062055 (+ 141us) server_negotiation.cc:407] Connection header received
0926 03:58:26.062108 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:26.062109 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:26.062117 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:26.062182 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:58:26.062331 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:26.062786 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:26.063646 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:26.063852 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:26.063934 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:26.064110 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:58:26.064141 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:58:26.064211 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:26.064265 (+ 54us) server_negotiation.cc:299] Negotiation successful
0926 03:58:26.064282 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 03:58:26.064389 2068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:26.061784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44309 (local address 127.0.98.3:44999)
0926 03:58:26.061981 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:26.061991 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:26.062052 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:26.062201 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:26.062202 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:26.062204 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 03:58:26.062255 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:26.062257 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:26.062869 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:26.062870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:26.063553 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:26.063555 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:26.063936 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:26.063936 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:26.063998 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:26.064019 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 03:58:26.064210 (+ 191us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 03:58:26.064212 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 03:58:26.064261 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 03:58:26.064304 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":88,"threads_started":1}
I20260926 03:58:26.066866 1666 raft_consensus.cc:1261] T 01ed46de90f045bcb28819854c98a90d P 266532c1f533453fb6df898015566b66 [term 1 FOLLOWER]: Refusing update from remote peer 55485c830e8a45e68078486b3b60ec47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:58:26.066912 1806 raft_consensus.cc:1261] T 01ed46de90f045bcb28819854c98a90d P 113a195864e74a88a6b10daacba06759 [term 1 FOLLOWER]: Refusing update from remote peer 55485c830e8a45e68078486b3b60ec47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:58:26.067111 2044 consensus_queue.cc:1035] T 01ed46de90f045bcb28819854c98a90d P 55485c830e8a45e68078486b3b60ec47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "266532c1f533453fb6df898015566b66" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:58:26.067245 2023 consensus_queue.cc:1035] T 01ed46de90f045bcb28819854c98a90d P 55485c830e8a45e68078486b3b60ec47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "113a195864e74a88a6b10daacba06759" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:58:26.075495 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:26.072465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54059 (local address 127.0.98.3:38109)
0926 03:58:26.072544 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:26.072546 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:26.072604 (+ 58us) server_negotiation.cc:407] Connection header received
0926 03:58:26.072730 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:26.072731 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:26.072740 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:26.072767 (+ 27us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:58:26.072933 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:26.073644 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:26.074748 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:26.075038 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:26.075076 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:26.075273 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:58:26.075304 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:58:26.075360 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:26.075427 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 03:58:26.075446 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 03:58:26.075549 2073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:26.072377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38109 (local address 127.0.98.1:54059)
0926 03:58:26.072540 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:26.072551 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:26.072655 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:26.072822 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:26.072823 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:26.072825 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 03:58:26.072865 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:26.072866 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:26.073739 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:26.073740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:26.074650 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:26.074652 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:26.075127 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:26.075128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:26.075180 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:26.075192 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 03:58:26.075375 (+ 183us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 03:58:26.075376 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 03:58:26.075421 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 03:58:26.075462 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":99,"threads_started":1}
I20260926 03:58:26.084717 1509 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54472:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:58:26.084820 1509 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54472:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:58:26.085441 1509 catalog_manager.cc:5782] T 00000000000000000000000000000000 P df5fd5aad81a453790d435063bdd82d4: Sending DeleteTablet for 3 replicas of tablet 01ed46de90f045bcb28819854c98a90d
I20260926 03:58:26.085645 1645 tablet_service.cc:1514] Processing DeleteTablet for tablet 01ed46de90f045bcb28819854c98a90d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:58:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58220
I20260926 03:58:26.085749 1786 tablet_service.cc:1514] Processing DeleteTablet for tablet 01ed46de90f045bcb28819854c98a90d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:58:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52074
I20260926 03:58:26.085940 2078 tablet_replica.cc:330] T 01ed46de90f045bcb28819854c98a90d P 113a195864e74a88a6b10daacba06759: stopping tablet replica
I20260926 03:58:26.085951 1926 tablet_service.cc:1514] Processing DeleteTablet for tablet 01ed46de90f045bcb28819854c98a90d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:58:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38124
I20260926 03:58:26.085999 2078 raft_consensus.cc:2229] T 01ed46de90f045bcb28819854c98a90d P 113a195864e74a88a6b10daacba06759 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:58:26.086064 2078 raft_consensus.cc:2258] T 01ed46de90f045bcb28819854c98a90d P 113a195864e74a88a6b10daacba06759 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:58:26.086330 2078 ts_tablet_manager.cc:1905] T 01ed46de90f045bcb28819854c98a90d P 113a195864e74a88a6b10daacba06759: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:58:26.086763 2077 tablet_replica.cc:330] T 01ed46de90f045bcb28819854c98a90d P 266532c1f533453fb6df898015566b66: stopping tablet replica
I20260926 03:58:26.086874 2077 raft_consensus.cc:2229] T 01ed46de90f045bcb28819854c98a90d P 266532c1f533453fb6df898015566b66 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:58:26.086966 2077 raft_consensus.cc:2258] T 01ed46de90f045bcb28819854c98a90d P 266532c1f533453fb6df898015566b66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:58:26.087235 2077 ts_tablet_manager.cc:1905] T 01ed46de90f045bcb28819854c98a90d P 266532c1f533453fb6df898015566b66: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:58:26.088073 2078 ts_tablet_manager.cc:1918] T 01ed46de90f045bcb28819854c98a90d P 113a195864e74a88a6b10daacba06759: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 03:58:26.088136 2078 log.cc:1196] T 01ed46de90f045bcb28819854c98a90d P 113a195864e74a88a6b10daacba06759: Deleting WAL directory at /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/minicluster-data/ts-1/wal/wals/01ed46de90f045bcb28819854c98a90d
I20260926 03:58:26.088366 2078 ts_tablet_manager.cc:1939] T 01ed46de90f045bcb28819854c98a90d P 113a195864e74a88a6b10daacba06759: Deleting consensus metadata
I20260926 03:58:26.088807 1495 catalog_manager.cc:4841] TS 113a195864e74a88a6b10daacba06759 (127.0.98.2:45311): tablet 01ed46de90f045bcb28819854c98a90d (table test-table [id=db13018cf24646989511433becbc4e0a]) successfully deleted
I20260926 03:58:26.089247 2077 ts_tablet_manager.cc:1918] T 01ed46de90f045bcb28819854c98a90d P 266532c1f533453fb6df898015566b66: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 03:58:26.089318 2077 log.cc:1196] T 01ed46de90f045bcb28819854c98a90d P 266532c1f533453fb6df898015566b66: Deleting WAL directory at /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/minicluster-data/ts-0/wal/wals/01ed46de90f045bcb28819854c98a90d
I20260926 03:58:26.089535 2077 ts_tablet_manager.cc:1939] T 01ed46de90f045bcb28819854c98a90d P 266532c1f533453fb6df898015566b66: Deleting consensus metadata
I20260926 03:58:26.090065 1494 catalog_manager.cc:4841] TS 266532c1f533453fb6df898015566b66 (127.0.98.1:44309): tablet 01ed46de90f045bcb28819854c98a90d (table test-table [id=db13018cf24646989511433becbc4e0a]) successfully deleted
I20260926 03:58:26.092756 2080 tablet_replica.cc:330] T 01ed46de90f045bcb28819854c98a90d P 55485c830e8a45e68078486b3b60ec47: stopping tablet replica
I20260926 03:58:26.092875 2080 raft_consensus.cc:2229] T 01ed46de90f045bcb28819854c98a90d P 55485c830e8a45e68078486b3b60ec47 [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:58:26.093010 2080 raft_consensus.cc:2258] T 01ed46de90f045bcb28819854c98a90d P 55485c830e8a45e68078486b3b60ec47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:58:26.093284 2080 ts_tablet_manager.cc:1905] T 01ed46de90f045bcb28819854c98a90d P 55485c830e8a45e68078486b3b60ec47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:58:26.095027 2080 ts_tablet_manager.cc:1918] T 01ed46de90f045bcb28819854c98a90d P 55485c830e8a45e68078486b3b60ec47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 03:58:26.095129 2080 log.cc:1196] T 01ed46de90f045bcb28819854c98a90d P 55485c830e8a45e68078486b3b60ec47: Deleting WAL directory at /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790395095474643-392-0/minicluster-data/ts-2/wal/wals/01ed46de90f045bcb28819854c98a90d
I20260926 03:58:26.095418 2080 ts_tablet_manager.cc:1939] T 01ed46de90f045bcb28819854c98a90d P 55485c830e8a45e68078486b3b60ec47: Deleting consensus metadata
I20260926 03:58:26.095786 1494 catalog_manager.cc:4841] TS 55485c830e8a45e68078486b3b60ec47 (127.0.98.3:38109): tablet 01ed46de90f045bcb28819854c98a90d (table test-table [id=db13018cf24646989511433becbc4e0a]) successfully deleted
Sep 26 03:58:26 dist-test-slave-nhpw krb5kdc[1464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395105, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 03:58:26.099872 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:26.086531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58236 (local address 127.0.98.1:44309)
0926 03:58:26.086568 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:26.086570 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:26.089796 (+ 3226us) server_negotiation.cc:407] Connection header received
0926 03:58:26.089886 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:26.089887 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:26.089898 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:26.089958 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:26.090729 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:26.091478 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:26.092517 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:26.092830 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:26.092871 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:26.096444 (+ 3573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:26.096450 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:26.096451 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:26.096458 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:26.098488 (+ 2030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:26.098885 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:26.098889 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:26.098889 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:26.098964 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:26.099171 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:26.099172 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:26.099172 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:26.099341 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:26.099473 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:26.099690 (+ 217us) server_negotiation.cc:299] Negotiation successful
0926 03:58:26.099772 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260926 03:58:26.100446 1706 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:58236
I20260926 03:58:26.108879 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:26.101365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54494 (local address 127.0.98.62:46765)
0926 03:58:26.101406 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:26.101408 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:26.101632 (+ 224us) server_negotiation.cc:407] Connection header received
0926 03:58:26.101666 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:26.101667 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:26.101678 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:26.101747 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:26.102425 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:26.103171 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:26.104197 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:26.104489 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:26.104571 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:26.105980 (+ 1409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:26.105982 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:26.105982 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:26.105989 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:26.107643 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:26.108031 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:26.108032 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:26.108033 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:26.108093 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:26.108312 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:26.108313 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:26.108314 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:26.108442 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:26.108546 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:26.108716 (+ 170us) server_negotiation.cc:299] Negotiation successful
0926 03:58:26.108792 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 03:58:26.109073 1509 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54494
I20260926 03:58:26.109843 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 1581
I20260926 03:58:26.114943 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 1721
I20260926 03:58:26.120105 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 1861
I20260926 03:58:26.125059 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 1477
2026-09-26T03:58:26Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5488 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:58:26 dist-test-slave-nhpw krb5kdc[2086](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:58:26 dist-test-slave-nhpw krb5kdc[2086](info): set up 2 sockets
Sep 26 03:58:26 dist-test-slave-nhpw krb5kdc[2086](info): commencing operation
krb5kdc: starting...
W20260926 03:58:27.156780 392 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/krb5kdc/test-user.keytab.
Sep 26 03:58:27 dist-test-slave-nhpw krb5kdc[2086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:58:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:58:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 03:58:27.272257 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42475
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42475
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:58:27.350814 2099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:58:27.351027 2099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:58:27.351048 2099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:58:27.352126 2099 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:58:27.352162 2099 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 03:58:27.352176 2099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:58:27.352190 2099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:58:27.352201 2099 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:58:27.352213 2099 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:58:27.353617 2099 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42475
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:58:27.353806 2099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:58:27.353982 2099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:58:27.358901 2104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:27.359057 2099 server_base.cc:1034] running on GCE node
W20260926 03:58:27.358906 2105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:27.358912 2107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:27.359383 2099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:58:27.359623 2099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:58:27.360787 2099 hybrid_clock.cc:648] HybridClock initialized: now 1790395107360771 us; error 31 us; skew 500 ppm
Sep 26 03:58:27 dist-test-slave-nhpw krb5kdc[2086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395107, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:58:27.363205 2099 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 03:58:27.363617 2099 webserver.cc:460] Webserver started at http://127.0.98.62:37233/ using document root <none> and password file <none>
I20260926 03:58:27.363833 2099 fs_manager.cc:359] Metadata directory not provided
I20260926 03:58:27.363901 2099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:58:27.364059 2099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:58:27.365206 2099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/minicluster-data/master-0/data/instance:
uuid: "d3414598d0e6464bb4b6f634282fb92a"
format_stamp: "Formatted at 2026-09-26 03:58:27 on dist-test-slave-nhpw"
server_key: "077d76c0c84d6c4fdcff37e58ea8e841"
server_key_iv: "40eb035e08ff5dce6e467164ceab83f2"
server_key_version: "encryptionkey@0"
I20260926 03:58:27.365635 2099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/minicluster-data/master-0/wal/instance:
uuid: "d3414598d0e6464bb4b6f634282fb92a"
format_stamp: "Formatted at 2026-09-26 03:58:27 on dist-test-slave-nhpw"
server_key: "077d76c0c84d6c4fdcff37e58ea8e841"
server_key_iv: "40eb035e08ff5dce6e467164ceab83f2"
server_key_version: "encryptionkey@0"
I20260926 03:58:27.367045 2099 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 03:58:27.367782 2114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:27.367962 2099 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 03:58:27.368060 2099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/minicluster-data/master-0/wal
uuid: "d3414598d0e6464bb4b6f634282fb92a"
format_stamp: "Formatted at 2026-09-26 03:58:27 on dist-test-slave-nhpw"
server_key: "077d76c0c84d6c4fdcff37e58ea8e841"
server_key_iv: "40eb035e08ff5dce6e467164ceab83f2"
server_key_version: "encryptionkey@0"
I20260926 03:58:27.368141 2099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:58:27.389315 2099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:58:27.391419 2099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:58:27.391585 2099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:58:27.396235 2099 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42475
I20260926 03:58:27.396281 2176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42475 every 8 connection(s)
I20260926 03:58:27.396616 2099 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
I20260926 03:58:27.397279 2177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:58:27.399632 2177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3414598d0e6464bb4b6f634282fb92a: Bootstrap starting.
I20260926 03:58:27.400244 2177 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3414598d0e6464bb4b6f634282fb92a: Neither blocks nor log segments found. Creating new log.
I20260926 03:58:27.400547 2177 log.cc:824] T 00000000000000000000000000000000 P d3414598d0e6464bb4b6f634282fb92a: Log is configured to *not* fsync() on all Append() calls
I20260926 03:58:27.401335 2177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3414598d0e6464bb4b6f634282fb92a: No bootstrap required, opened a new log
I20260926 03:58:27.403457 2177 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3414598d0e6464bb4b6f634282fb92a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3414598d0e6464bb4b6f634282fb92a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42475 } }
I20260926 03:58:27.403623 2177 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3414598d0e6464bb4b6f634282fb92a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:58:27.403669 2177 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3414598d0e6464bb4b6f634282fb92a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3414598d0e6464bb4b6f634282fb92a, State: Initialized, Role: FOLLOWER
I20260926 03:58:27.403846 2177 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3414598d0e6464bb4b6f634282fb92a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3414598d0e6464bb4b6f634282fb92a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42475 } }
I20260926 03:58:27.403930 2177 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3414598d0e6464bb4b6f634282fb92a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:58:27.403971 2177 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3414598d0e6464bb4b6f634282fb92a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:58:27.404021 2177 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3414598d0e6464bb4b6f634282fb92a [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:58:27.404563 2177 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3414598d0e6464bb4b6f634282fb92a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3414598d0e6464bb4b6f634282fb92a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42475 } }
I20260926 03:58:27.404716 2177 leader_election.cc:302] T 00000000000000000000000000000000 P d3414598d0e6464bb4b6f634282fb92a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3414598d0e6464bb4b6f634282fb92a; no voters:
I20260926 03:58:27.404892 2177 leader_election.cc:288] T 00000000000000000000000000000000 P d3414598d0e6464bb4b6f634282fb92a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:58:27.404953 2180 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3414598d0e6464bb4b6f634282fb92a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:58:27.405131 2180 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3414598d0e6464bb4b6f634282fb92a [term 1 LEADER]: Becoming Leader. State: Replica: d3414598d0e6464bb4b6f634282fb92a, State: Running, Role: LEADER
I20260926 03:58:27.405169 2177 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3414598d0e6464bb4b6f634282fb92a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:58:27.405217 2180 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3414598d0e6464bb4b6f634282fb92a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3414598d0e6464bb4b6f634282fb92a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42475 } }
I20260926 03:58:27.405644 2182 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3414598d0e6464bb4b6f634282fb92a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3414598d0e6464bb4b6f634282fb92a. Latest consensus state: current_term: 1 leader_uuid: "d3414598d0e6464bb4b6f634282fb92a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3414598d0e6464bb4b6f634282fb92a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42475 } } }
I20260926 03:58:27.405750 2182 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3414598d0e6464bb4b6f634282fb92a [sys.catalog]: This master's current role is: LEADER
I20260926 03:58:27.405628 2181 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3414598d0e6464bb4b6f634282fb92a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3414598d0e6464bb4b6f634282fb92a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3414598d0e6464bb4b6f634282fb92a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42475 } } }
I20260926 03:58:27.406183 2189 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:58:27.406191 2181 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3414598d0e6464bb4b6f634282fb92a [sys.catalog]: This master's current role is: LEADER
I20260926 03:58:27.406740 2189 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:58:27.406745 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 2099
I20260926 03:58:27.406836 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/minicluster-data/master-0/wal/instance
I20260926 03:58:27.407016 392 external_mini_cluster.cc:1428] Setting key 2d575ceae2674665f6d51dcfa482c26b
I20260926 03:58:27.408577 2189 catalog_manager.cc:1347] Generated new cluster ID: b9c2573b725c4ddca4f3d22624163240
I20260926 03:58:27.408649 2189 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 03:58:27 dist-test-slave-nhpw krb5kdc[2086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 03:58:27.419458 2199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:27.407648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43362 (local address 127.0.98.62:42475)
0926 03:58:27.407820 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:27.407830 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:27.407860 (+ 30us) server_negotiation.cc:407] Connection header received
0926 03:58:27.407994 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:27.408001 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:27.408072 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:27.408233 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:27.409038 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:27.410465 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:27.411562 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:27.412199 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:27.412305 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:27.415368 (+ 3063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:27.415373 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:27.415373 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:27.415385 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:27.417357 (+ 1972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:27.417757 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:27.417761 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:27.417761 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:27.417892 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:27.418107 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:27.418108 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:27.418108 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:27.418413 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:27.418562 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:27.419083 (+ 521us) server_negotiation.cc:299] Negotiation successful
0926 03:58:27.419219 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":130,"thread_start_us":86,"threads_started":1}
I20260926 03:58:27.419811 2189 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:58:27.420401 2189 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:58:27.422571 2189 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3414598d0e6464bb4b6f634282fb92a: Generated new TSK 0
I20260926 03:58:27.422752 2189 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 03:58:27.456245 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42475
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:58:27.536537 2203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:58:27.536751 2203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:58:27.536775 2203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:58:27.537921 2203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:58:27.537962 2203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:58:27.537982 2203 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 03:58:27.538017 2203 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 03:58:27.538053 2203 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 03:58:27.539463 2203 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:42475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2203
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:58:27.539702 2203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:58:27.539924 2203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:58:27.545086 2209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:27.545065 2211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:27.545148 2208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:27.545321 2203 server_base.cc:1034] running on GCE node
I20260926 03:58:27.545521 2203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:58:27.545708 2203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:58:27.546856 2203 hybrid_clock.cc:648] HybridClock initialized: now 1790395107546842 us; error 29 us; skew 500 ppm
Sep 26 03:58:27 dist-test-slave-nhpw krb5kdc[2086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395107, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:58:27.549309 2203 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 03:58:27.549794 2203 webserver.cc:460] Webserver started at http://127.0.98.1:42841/ using document root <none> and password file <none>
I20260926 03:58:27.550037 2203 fs_manager.cc:359] Metadata directory not provided
I20260926 03:58:27.550081 2203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:58:27.550225 2203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:58:27.551295 2203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/minicluster-data/ts-0/data/instance:
uuid: "216d653228874c8bb95fa8cfdda70c04"
format_stamp: "Formatted at 2026-09-26 03:58:27 on dist-test-slave-nhpw"
server_key: "922b455ea3873815e009d379912ba936"
server_key_iv: "98490fc738e698fc63f389d48d2aedf8"
server_key_version: "encryptionkey@0"
I20260926 03:58:27.551656 2203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/minicluster-data/ts-0/wal/instance:
uuid: "216d653228874c8bb95fa8cfdda70c04"
format_stamp: "Formatted at 2026-09-26 03:58:27 on dist-test-slave-nhpw"
server_key: "922b455ea3873815e009d379912ba936"
server_key_iv: "98490fc738e698fc63f389d48d2aedf8"
server_key_version: "encryptionkey@0"
I20260926 03:58:27.553022 2203 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 03:58:27.553761 2218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:27.553934 2203 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 03:58:27.554036 2203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/minicluster-data/ts-0/wal
uuid: "216d653228874c8bb95fa8cfdda70c04"
format_stamp: "Formatted at 2026-09-26 03:58:27 on dist-test-slave-nhpw"
server_key: "922b455ea3873815e009d379912ba936"
server_key_iv: "98490fc738e698fc63f389d48d2aedf8"
server_key_version: "encryptionkey@0"
I20260926 03:58:27.554107 2203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:58:27.566792 2203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:58:27.569154 2203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:58:27.569306 2203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:58:27.569775 2203 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:58:27.569816 2203 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:27.569839 2203 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:58:27.569850 2203 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:27.575582 2203 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33193
I20260926 03:58:27.575834 2333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33193 every 8 connection(s)
I20260926 03:58:27.576030 2203 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
Sep 26 03:58:27 dist-test-slave-nhpw krb5kdc[2086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395107, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 03:58:27.579994 2199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:27.570253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38709 (local address 127.0.98.62:42475)
0926 03:58:27.570296 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:27.570298 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:27.570551 (+ 253us) server_negotiation.cc:407] Connection header received
0926 03:58:27.571150 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:27.571151 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:27.571162 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:27.571219 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:27.572074 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:27.572918 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:27.574412 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:27.574787 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:27.574861 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:27.577460 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:27.577461 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:27.577462 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:27.577471 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:27.578895 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:27.579257 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:27.579258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:27.579258 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:27.579305 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:27.579511 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:27.579511 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:27.579511 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:27.579604 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:27.579653 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:27.579873 (+ 220us) server_negotiation.cc:299] Negotiation successful
0926 03:58:27.579934 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 03:58:27.580181 2227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:27.570247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42475 (local address 127.0.98.1:38709)
0926 03:58:27.570428 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:27.570470 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:27.571044 (+ 574us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:27.571311 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:27.571315 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:27.571684 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:27.571981 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:27.571986 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:27.573009 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:27.573010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:27.574318 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:27.574320 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:27.574930 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:27.574931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:27.575078 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:27.575600 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:27.575612 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:27.577368 (+ 1756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:58:27.578971 (+ 1603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:27.578976 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:27.578977 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:27.579183 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:27.579361 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:27.579362 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:27.579362 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:27.579454 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:27.579661 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:58:27.579665 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:58:27.579775 (+ 110us) client_negotiation.cc:769] Sending connection context
0926 03:58:27.579860 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 03:58:27.579982 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":104,"threads_started":1}
I20260926 03:58:27.580742 2334 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42475
I20260926 03:58:27.580883 2334 heartbeater.cc:464] Registering TS with master...
I20260926 03:58:27.580981 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 2203
I20260926 03:58:27.581079 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/minicluster-data/ts-0/wal/instance
I20260926 03:58:27.581086 2334 heartbeater.cc:511] Master 127.0.98.62:42475 requested a full tablet report, sending...
I20260926 03:58:27.581276 392 external_mini_cluster.cc:1428] Setting key b8016f7489ad123fca23f953bb01831c
I20260926 03:58:27.581655 2130 ts_manager.cc:194] Registered new tserver with Master: 216d653228874c8bb95fa8cfdda70c04 (127.0.98.1:33193)
I20260926 03:58:27.582489 2130 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38709
I20260926 03:58:27.587699 2199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:27.581848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40307 (local address 127.0.98.62:42475)
0926 03:58:27.581907 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:27.581909 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:27.582139 (+ 230us) server_negotiation.cc:407] Connection header received
0926 03:58:27.582185 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:27.582186 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:27.582196 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:27.582261 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:27.582622 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:27.583360 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:27.584040 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:27.584281 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:27.584341 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:27.585252 (+ 911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:27.585253 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:27.585254 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:27.585260 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:27.586597 (+ 1337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:27.586940 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:27.586941 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:27.586941 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:27.586992 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:27.587197 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:27.587198 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:27.587199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:27.587304 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:27.587382 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:27.587534 (+ 152us) server_negotiation.cc:299] Negotiation successful
0926 03:58:27.587607 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 03:58:27.587713 2340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:27.581892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42475 (local address 127.0.98.1:40307)
0926 03:58:27.582053 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:27.582065 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:27.582131 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:27.582253 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:27.582254 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:27.582467 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:27.582531 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:27.582533 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:27.583474 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:27.583475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:27.583959 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:27.583960 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:27.584359 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:27.584359 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:27.584410 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:27.584871 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:27.584876 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:27.585141 (+ 265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:58:27.586698 (+ 1557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:27.586699 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:27.586699 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:27.586869 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:27.587069 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:27.587070 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:27.587071 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:27.587128 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:27.587373 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:58:27.587374 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:58:27.587422 (+ 48us) client_negotiation.cc:769] Sending connection context
0926 03:58:27.587509 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 03:58:27.587607 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":95,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 03:58:27.607692 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42475
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:58:27.689018 2343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:58:27.689218 2343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:58:27.689242 2343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:58:27.690313 2343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:58:27.690347 2343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:58:27.690366 2343 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 03:58:27.690378 2343 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 03:58:27.690394 2343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 03:58:27.691805 2343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:42475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2343
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:58:27.691978 2343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:58:27.692148 2343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:58:27.697079 2348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:27.697211 2343 server_base.cc:1034] running on GCE node
W20260926 03:58:27.697060 2349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:27.697059 2351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:27.697497 2343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:58:27.697690 2343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:58:27.698848 2343 hybrid_clock.cc:648] HybridClock initialized: now 1790395107698836 us; error 28 us; skew 500 ppm
Sep 26 03:58:27 dist-test-slave-nhpw krb5kdc[2086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395107, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:58:27.701278 2343 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 03:58:27.701683 2343 webserver.cc:460] Webserver started at http://127.0.98.2:43695/ using document root <none> and password file <none>
I20260926 03:58:27.701897 2343 fs_manager.cc:359] Metadata directory not provided
I20260926 03:58:27.701960 2343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:58:27.702113 2343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:58:27.703114 2343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/minicluster-data/ts-1/data/instance:
uuid: "830c2976e7ef4eb4a96416333fcf1aa1"
format_stamp: "Formatted at 2026-09-26 03:58:27 on dist-test-slave-nhpw"
server_key: "87fa67f50a98c15e770939105d62d95a"
server_key_iv: "f1cb9838f0e642629e91bcf41d9f648d"
server_key_version: "encryptionkey@0"
I20260926 03:58:27.703460 2343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/minicluster-data/ts-1/wal/instance:
uuid: "830c2976e7ef4eb4a96416333fcf1aa1"
format_stamp: "Formatted at 2026-09-26 03:58:27 on dist-test-slave-nhpw"
server_key: "87fa67f50a98c15e770939105d62d95a"
server_key_iv: "f1cb9838f0e642629e91bcf41d9f648d"
server_key_version: "encryptionkey@0"
I20260926 03:58:27.704811 2343 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 03:58:27.705571 2358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:27.705732 2343 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 03:58:27.705827 2343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/minicluster-data/ts-1/wal
uuid: "830c2976e7ef4eb4a96416333fcf1aa1"
format_stamp: "Formatted at 2026-09-26 03:58:27 on dist-test-slave-nhpw"
server_key: "87fa67f50a98c15e770939105d62d95a"
server_key_iv: "f1cb9838f0e642629e91bcf41d9f648d"
server_key_version: "encryptionkey@0"
I20260926 03:58:27.705905 2343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:58:27.715577 2343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:58:27.717796 2343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:58:27.717952 2343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:58:27.718415 2343 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:58:27.718475 2343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:27.718509 2343 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:58:27.718539 2343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 03:58:27 dist-test-slave-nhpw krb5kdc[2086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395107, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 03:58:27.725644 2343 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44451
I20260926 03:58:27.725705 2473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44451 every 8 connection(s)
I20260926 03:58:27.726040 2343 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
I20260926 03:58:27.728381 2199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:27.718868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52783 (local address 127.0.98.62:42475)
0926 03:58:27.718943 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:27.718946 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:27.719134 (+ 188us) server_negotiation.cc:407] Connection header received
0926 03:58:27.719580 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:27.719582 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:27.719595 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:27.719681 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:27.720478 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:27.721315 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:27.722466 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:27.722788 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:27.722867 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:27.725546 (+ 2679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:27.725547 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:27.725548 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:27.725557 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:27.727053 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:27.727453 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:27.727455 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:27.727455 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:27.727512 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:27.727765 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:27.727765 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:27.727766 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:27.727880 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:27.727960 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:27.728217 (+ 257us) server_negotiation.cc:299] Negotiation successful
0926 03:58:27.728292 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 03:58:27.728564 2367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:27.718844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42475 (local address 127.0.98.2:52783)
0926 03:58:27.719001 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:27.719039 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:27.719486 (+ 447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:27.719729 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:27.719732 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:27.720086 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:27.720359 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:27.720366 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:27.721440 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:27.721442 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:27.722375 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:27.722377 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:27.722886 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:27.722887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:27.723048 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:27.723600 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:27.723611 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:27.725465 (+ 1854us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:58:27.727151 (+ 1686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:27.727156 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:27.727158 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:27.727372 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:27.727592 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:27.727593 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:27.727593 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:27.727697 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:27.727950 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:58:27.727955 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:58:27.728088 (+ 133us) client_negotiation.cc:769] Sending connection context
0926 03:58:27.728195 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 03:58:27.728341 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":95,"threads_started":1}
I20260926 03:58:27.729012 2474 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42475
I20260926 03:58:27.729090 2474 heartbeater.cc:464] Registering TS with master...
I20260926 03:58:27.729269 2474 heartbeater.cc:511] Master 127.0.98.62:42475 requested a full tablet report, sending...
I20260926 03:58:27.729612 2131 ts_manager.cc:194] Registered new tserver with Master: 830c2976e7ef4eb4a96416333fcf1aa1 (127.0.98.2:44451)
I20260926 03:58:27.730121 2131 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:52783
I20260926 03:58:27.732375 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 2343
I20260926 03:58:27.732477 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/minicluster-data/ts-1/wal/instance
I20260926 03:58:27.732679 392 external_mini_cluster.cc:1428] Setting key add04ddf20b2eb745d23133a7748f370
I20260926 03:58:27.735412 2199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:27.730035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34249 (local address 127.0.98.62:42475)
0926 03:58:27.730072 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:27.730073 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:27.730144 (+ 71us) server_negotiation.cc:407] Connection header received
0926 03:58:27.730209 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:27.730210 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:27.730220 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:27.730278 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:27.730570 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:27.731154 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:27.731819 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:27.732005 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:27.732053 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:27.733037 (+ 984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:27.733038 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:27.733038 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:27.733045 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:27.734292 (+ 1247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:27.734625 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:27.734626 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:27.734626 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:27.734677 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:27.734886 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:27.734887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:27.734887 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:27.735028 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:27.735106 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:27.735258 (+ 152us) server_negotiation.cc:299] Negotiation successful
0926 03:58:27.735324 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 03:58:27.735431 2480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:27.729947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42475 (local address 127.0.98.2:34249)
0926 03:58:27.730081 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:27.730092 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:27.730153 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:27.730283 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:27.730284 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:27.730447 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:27.730500 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:27.730501 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:27.731251 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:27.731252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:27.731752 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:27.731753 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:27.732068 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:27.732069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:27.732121 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:27.732544 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:27.732549 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:27.732965 (+ 416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:58:27.734385 (+ 1420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:27.734386 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:27.734386 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:27.734553 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:27.734762 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:27.734763 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:27.734764 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:27.734819 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:27.735099 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:58:27.735099 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:58:27.735142 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 03:58:27.735230 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 03:58:27.735324 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":86,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 03:58:27.756136 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42475
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:58:27.835724 2483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:58:27.835955 2483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:58:27.835980 2483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:58:27.837205 2483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:58:27.837249 2483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:58:27.837270 2483 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 03:58:27.837282 2483 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 03:58:27.837299 2483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 03:58:27.838730 2483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:42475
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:58:27.838971 2483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:58:27.839223 2483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:58:27.844169 2488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:27.844183 2491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:27.844180 2489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:27.844427 2483 server_base.cc:1034] running on GCE node
I20260926 03:58:27.844673 2483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:58:27.844933 2483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:58:27.846158 2483 hybrid_clock.cc:648] HybridClock initialized: now 1790395107846149 us; error 40 us; skew 500 ppm
Sep 26 03:58:27 dist-test-slave-nhpw krb5kdc[2086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395107, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:58:27.848690 2483 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 03:58:27.849176 2483 webserver.cc:460] Webserver started at http://127.0.98.3:37143/ using document root <none> and password file <none>
I20260926 03:58:27.849421 2483 fs_manager.cc:359] Metadata directory not provided
I20260926 03:58:27.849490 2483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:58:27.849632 2483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:58:27.850680 2483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/minicluster-data/ts-2/data/instance:
uuid: "553c7127b2b446729881acd33d89da44"
format_stamp: "Formatted at 2026-09-26 03:58:27 on dist-test-slave-nhpw"
server_key: "365e56e05fbc0169c0744837f530d280"
server_key_iv: "accbe8e4a5d327e9f226474c9cedafbe"
server_key_version: "encryptionkey@0"
I20260926 03:58:27.851064 2483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/minicluster-data/ts-2/wal/instance:
uuid: "553c7127b2b446729881acd33d89da44"
format_stamp: "Formatted at 2026-09-26 03:58:27 on dist-test-slave-nhpw"
server_key: "365e56e05fbc0169c0744837f530d280"
server_key_iv: "accbe8e4a5d327e9f226474c9cedafbe"
server_key_version: "encryptionkey@0"
I20260926 03:58:27.852349 2483 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 03:58:27.853170 2498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:27.853366 2483 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 03:58:27.853472 2483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/minicluster-data/ts-2/wal
uuid: "553c7127b2b446729881acd33d89da44"
format_stamp: "Formatted at 2026-09-26 03:58:27 on dist-test-slave-nhpw"
server_key: "365e56e05fbc0169c0744837f530d280"
server_key_iv: "accbe8e4a5d327e9f226474c9cedafbe"
server_key_version: "encryptionkey@0"
I20260926 03:58:27.853560 2483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:58:27.866258 2483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:58:27.868516 2483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:58:27.868806 2483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:58:27.869357 2483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:58:27.869421 2483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:27.869460 2483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:58:27.869482 2483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 03:58:27 dist-test-slave-nhpw krb5kdc[2086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395107, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 03:58:27.877271 2483 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35577
I20260926 03:58:27.877331 2613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35577 every 8 connection(s)
I20260926 03:58:27.877626 2483 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
I20260926 03:58:27.879923 2199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:27.869742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57589 (local address 127.0.98.62:42475)
0926 03:58:27.869809 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:27.869812 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:27.870082 (+ 270us) server_negotiation.cc:407] Connection header received
0926 03:58:27.870701 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:27.870702 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:27.870714 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:27.870770 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:27.871487 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:27.872104 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:27.873308 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:27.873526 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:27.873613 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:27.877061 (+ 3448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:27.877063 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:27.877064 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:27.877073 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:27.878702 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:27.879071 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:27.879072 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:27.879073 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:27.879123 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:27.879333 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:27.879333 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:27.879333 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:27.879430 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:27.879497 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:27.879794 (+ 297us) server_negotiation.cc:299] Negotiation successful
0926 03:58:27.879861 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 03:58:27.880177 2507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:27.869749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42475 (local address 127.0.98.3:57589)
0926 03:58:27.869935 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:27.869975 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:27.870592 (+ 617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:27.870841 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:27.870846 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:27.871160 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:27.871396 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:27.871402 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:27.872210 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:27.872211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:27.873217 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:27.873220 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:27.873623 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:27.873624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:27.873760 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:27.874302 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:27.874312 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:27.876928 (+ 2616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:58:27.878787 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:27.878793 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:27.878794 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:27.879001 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:27.879182 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:27.879183 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:27.879183 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:27.879279 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:27.879491 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:58:27.879494 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:58:27.879600 (+ 106us) client_negotiation.cc:769] Sending connection context
0926 03:58:27.879781 (+ 181us) client_negotiation.cc:240] Negotiation successful
0926 03:58:27.879919 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":96,"threads_started":1}
I20260926 03:58:27.880704 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 2483
I20260926 03:58:27.880735 2614 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42475
I20260926 03:58:27.880800 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/minicluster-data/ts-2/wal/instance
I20260926 03:58:27.880860 2614 heartbeater.cc:464] Registering TS with master...
I20260926 03:58:27.881031 392 external_mini_cluster.cc:1428] Setting key 1c747cca75962b43ea5e621ddf1af8aa
I20260926 03:58:27.881043 2614 heartbeater.cc:511] Master 127.0.98.62:42475 requested a full tablet report, sending...
I20260926 03:58:27.881376 2130 ts_manager.cc:194] Registered new tserver with Master: 553c7127b2b446729881acd33d89da44 (127.0.98.3:35577)
I20260926 03:58:27.881872 2130 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57589
I20260926 03:58:27.881934 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:58:27.889055 2199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:27.881901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33089 (local address 127.0.98.62:42475)
0926 03:58:27.883525 (+ 1624us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:27.883526 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:27.883534 (+ 8us) server_negotiation.cc:407] Connection header received
0926 03:58:27.883551 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:27.883552 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:27.883559 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:27.883643 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:27.883960 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:27.884432 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:27.885239 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:27.885444 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:27.885530 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:27.886453 (+ 923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:27.886455 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:27.886455 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:27.886462 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:27.887877 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:27.888250 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:27.888251 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:27.888252 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:27.888305 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:27.888521 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:27.888522 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:27.888522 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:27.888627 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:27.888730 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:27.888893 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 03:58:27.888965 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1605}
I20260926 03:58:27.889068 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:27.881860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42475 (local address 127.0.98.3:33089)
0926 03:58:27.881999 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:27.882010 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:27.882058 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:27.883650 (+ 1592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:27.883651 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:27.883832 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:27.883878 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:27.883879 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:27.884529 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:27.884530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:27.885136 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:27.885138 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:27.885531 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:27.885532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:27.885605 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:27.886062 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:27.886068 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:27.886350 (+ 282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:58:27.887983 (+ 1633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:27.887984 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:27.887985 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:27.888157 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:27.888380 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:27.888381 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:27.888381 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:27.888438 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:27.888721 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:58:27.888722 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:58:27.888766 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 03:58:27.888866 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 03:58:27.888966 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":85,"threads_started":1}
Sep 26 03:58:27 dist-test-slave-nhpw krb5kdc[2086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 03:58:27 dist-test-slave-nhpw krb5kdc[2086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 03:58:27.905995 2199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:27.897869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43364 (local address 127.0.98.62:42475)
0926 03:58:27.898001 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:27.898003 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:27.898188 (+ 185us) server_negotiation.cc:407] Connection header received
0926 03:58:27.898305 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:27.898306 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:27.898316 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:27.898388 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:27.898977 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:27.899633 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:27.900389 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:27.900591 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:27.900704 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:27.903096 (+ 2392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:27.903098 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:27.903099 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:27.903106 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:27.904583 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:27.905010 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:27.905011 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:27.905012 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:27.905069 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:27.905272 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:27.905273 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:27.905274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:27.905386 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:27.905523 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:27.905613 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 03:58:27.905863 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260926 03:58:27.907706 2130 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43364:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:58:27.909440 2130 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:58:27.916069 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:27.911726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33193 (local address 127.0.0.1:42872)
0926 03:58:27.911898 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:27.911939 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:27.912008 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:27.912504 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:27.912505 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:27.912506 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:58:27.912556 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:27.912558 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:27.913878 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:27.913879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:27.914821 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:27.914823 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:27.915754 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:27.915755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:27.915849 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:27.915862 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 03:58:27.915911 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 03:58:27.915959 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":60,"threads_started":1}
I20260926 03:58:27.916180 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:27.911932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44451 (local address 127.0.0.1:55982)
0926 03:58:27.912350 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:27.912361 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:27.912429 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:27.912866 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:27.912867 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:27.912870 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:58:27.912908 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:27.912909 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:27.914101 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:27.914102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:27.915151 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:27.915153 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:27.915916 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:27.915917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:27.915973 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:27.915976 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 03:58:27.916033 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 03:58:27.916067 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":51,"threads_started":1}
I20260926 03:58:27.916253 2628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:27.911726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35577 (local address 127.0.0.1:59398)
0926 03:58:27.912029 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:27.912038 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:27.912080 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:27.912311 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:27.912315 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:27.912317 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:58:27.912391 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:27.912392 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:27.913590 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:27.913592 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:27.914846 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:27.914848 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:27.915746 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:27.915747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:27.915862 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:27.915867 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 03:58:27.915934 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 03:58:27.915980 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":2,"thread_start_us":57,"threads_started":1}
I20260926 03:58:27.916410 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:27.911853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42872 (local address 127.0.98.1:33193)
0926 03:58:27.912329 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:27.912334 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:27.912343 (+ 9us) server_negotiation.cc:407] Connection header received
0926 03:58:27.912383 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:27.912385 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:27.912400 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:27.912505 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:58:27.912669 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:27.913779 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:27.914924 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:27.915653 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:27.916191 (+ 538us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:27.916258 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:27.916284 (+ 26us) server_negotiation.cc:299] Negotiation successful
0926 03:58:27.916324 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":105,"threads_started":1}
I20260926 03:58:27.916452 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:27.911917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59398 (local address 127.0.98.3:35577)
0926 03:58:27.912091 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:27.912096 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:27.912108 (+ 12us) server_negotiation.cc:407] Connection header received
0926 03:58:27.912155 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:27.912157 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:27.912175 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:27.912267 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:58:27.912511 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:27.913498 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:27.914947 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:27.915653 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:27.915731 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:27.915804 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:27.916112 (+ 308us) server_negotiation.cc:299] Negotiation successful
0926 03:58:27.916158 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":75,"threads_started":1}
I20260926 03:58:27.917156 2266 tablet_service.cc:1467] Processing CreateTablet for tablet fe3f2fc004124efabd73cbef56710ac9 (DEFAULT_TABLE table=test-table [id=61eebd3e319541018adc5d24d7865279]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:58:27.917162 2548 tablet_service.cc:1467] Processing CreateTablet for tablet fe3f2fc004124efabd73cbef56710ac9 (DEFAULT_TABLE table=test-table [id=61eebd3e319541018adc5d24d7865279]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:58:27.917497 2548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe3f2fc004124efabd73cbef56710ac9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:58:27.917506 2266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe3f2fc004124efabd73cbef56710ac9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:58:27.919895 2634 tablet_bootstrap.cc:492] T fe3f2fc004124efabd73cbef56710ac9 P 553c7127b2b446729881acd33d89da44: Bootstrap starting.
I20260926 03:58:27.920681 2634 tablet_bootstrap.cc:654] T fe3f2fc004124efabd73cbef56710ac9 P 553c7127b2b446729881acd33d89da44: Neither blocks nor log segments found. Creating new log.
I20260926 03:58:27.920707 2635 tablet_bootstrap.cc:492] T fe3f2fc004124efabd73cbef56710ac9 P 216d653228874c8bb95fa8cfdda70c04: Bootstrap starting.
I20260926 03:58:27.920969 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:27.912124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55982 (local address 127.0.98.2:44451)
0926 03:58:27.912333 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:27.912338 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:27.912611 (+ 273us) server_negotiation.cc:407] Connection header received
0926 03:58:27.912681 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:27.912683 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:27.912700 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:27.912799 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:58:27.912997 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:27.914013 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:27.915239 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:27.915827 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:27.915916 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:27.915978 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:27.916062 (+ 84us) server_negotiation.cc:299] Negotiation successful
0926 03:58:27.916108 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":86,"threads_started":1}
I20260926 03:58:27.920998 2634 log.cc:824] T fe3f2fc004124efabd73cbef56710ac9 P 553c7127b2b446729881acd33d89da44: Log is configured to *not* fsync() on all Append() calls
I20260926 03:58:27.921411 2635 tablet_bootstrap.cc:654] T fe3f2fc004124efabd73cbef56710ac9 P 216d653228874c8bb95fa8cfdda70c04: Neither blocks nor log segments found. Creating new log.
I20260926 03:58:27.921680 2634 tablet_bootstrap.cc:492] T fe3f2fc004124efabd73cbef56710ac9 P 553c7127b2b446729881acd33d89da44: No bootstrap required, opened a new log
I20260926 03:58:27.921762 2634 ts_tablet_manager.cc:1397] T fe3f2fc004124efabd73cbef56710ac9 P 553c7127b2b446729881acd33d89da44: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 03:58:27.921643 2408 tablet_service.cc:1467] Processing CreateTablet for tablet fe3f2fc004124efabd73cbef56710ac9 (DEFAULT_TABLE table=test-table [id=61eebd3e319541018adc5d24d7865279]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:58:27.921761 2635 log.cc:824] T fe3f2fc004124efabd73cbef56710ac9 P 216d653228874c8bb95fa8cfdda70c04: Log is configured to *not* fsync() on all Append() calls
I20260926 03:58:27.922003 2408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe3f2fc004124efabd73cbef56710ac9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:58:27.922580 2635 tablet_bootstrap.cc:492] T fe3f2fc004124efabd73cbef56710ac9 P 216d653228874c8bb95fa8cfdda70c04: No bootstrap required, opened a new log
I20260926 03:58:27.922660 2635 ts_tablet_manager.cc:1397] T fe3f2fc004124efabd73cbef56710ac9 P 216d653228874c8bb95fa8cfdda70c04: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 03:58:27.923595 2634 raft_consensus.cc:348] T fe3f2fc004124efabd73cbef56710ac9 P 553c7127b2b446729881acd33d89da44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "553c7127b2b446729881acd33d89da44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35577 } } peers { permanent_uuid: "216d653228874c8bb95fa8cfdda70c04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33193 } } peers { permanent_uuid: "830c2976e7ef4eb4a96416333fcf1aa1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44451 } }
I20260926 03:58:27.923744 2634 raft_consensus.cc:374] T fe3f2fc004124efabd73cbef56710ac9 P 553c7127b2b446729881acd33d89da44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:58:27.923785 2634 raft_consensus.cc:729] T fe3f2fc004124efabd73cbef56710ac9 P 553c7127b2b446729881acd33d89da44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 553c7127b2b446729881acd33d89da44, State: Initialized, Role: FOLLOWER
I20260926 03:58:27.923895 2634 consensus_queue.cc:260] T fe3f2fc004124efabd73cbef56710ac9 P 553c7127b2b446729881acd33d89da44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "553c7127b2b446729881acd33d89da44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35577 } } peers { permanent_uuid: "216d653228874c8bb95fa8cfdda70c04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33193 } } peers { permanent_uuid: "830c2976e7ef4eb4a96416333fcf1aa1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44451 } }
I20260926 03:58:27.924178 2634 ts_tablet_manager.cc:1428] T fe3f2fc004124efabd73cbef56710ac9 P 553c7127b2b446729881acd33d89da44: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 03:58:27.924746 2614 heartbeater.cc:503] Master 127.0.98.62:42475 was elected leader, sending a full tablet report...
I20260926 03:58:27.925490 2635 raft_consensus.cc:348] T fe3f2fc004124efabd73cbef56710ac9 P 216d653228874c8bb95fa8cfdda70c04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "553c7127b2b446729881acd33d89da44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35577 } } peers { permanent_uuid: "216d653228874c8bb95fa8cfdda70c04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33193 } } peers { permanent_uuid: "830c2976e7ef4eb4a96416333fcf1aa1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44451 } }
I20260926 03:58:27.925663 2635 raft_consensus.cc:374] T fe3f2fc004124efabd73cbef56710ac9 P 216d653228874c8bb95fa8cfdda70c04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:58:27.925707 2635 raft_consensus.cc:729] T fe3f2fc004124efabd73cbef56710ac9 P 216d653228874c8bb95fa8cfdda70c04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 216d653228874c8bb95fa8cfdda70c04, State: Initialized, Role: FOLLOWER
I20260926 03:58:27.925860 2635 consensus_queue.cc:260] T fe3f2fc004124efabd73cbef56710ac9 P 216d653228874c8bb95fa8cfdda70c04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "553c7127b2b446729881acd33d89da44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35577 } } peers { permanent_uuid: "216d653228874c8bb95fa8cfdda70c04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33193 } } peers { permanent_uuid: "830c2976e7ef4eb4a96416333fcf1aa1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44451 } }
I20260926 03:58:27.925935 2639 tablet_bootstrap.cc:492] T fe3f2fc004124efabd73cbef56710ac9 P 830c2976e7ef4eb4a96416333fcf1aa1: Bootstrap starting.
I20260926 03:58:27.926134 2635 ts_tablet_manager.cc:1428] T fe3f2fc004124efabd73cbef56710ac9 P 216d653228874c8bb95fa8cfdda70c04: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 03:58:27.926167 2334 heartbeater.cc:503] Master 127.0.98.62:42475 was elected leader, sending a full tablet report...
I20260926 03:58:27.926540 2639 tablet_bootstrap.cc:654] T fe3f2fc004124efabd73cbef56710ac9 P 830c2976e7ef4eb4a96416333fcf1aa1: Neither blocks nor log segments found. Creating new log.
I20260926 03:58:27.926834 2639 log.cc:824] T fe3f2fc004124efabd73cbef56710ac9 P 830c2976e7ef4eb4a96416333fcf1aa1: Log is configured to *not* fsync() on all Append() calls
I20260926 03:58:27.927582 2639 tablet_bootstrap.cc:492] T fe3f2fc004124efabd73cbef56710ac9 P 830c2976e7ef4eb4a96416333fcf1aa1: No bootstrap required, opened a new log
I20260926 03:58:27.927644 2639 ts_tablet_manager.cc:1397] T fe3f2fc004124efabd73cbef56710ac9 P 830c2976e7ef4eb4a96416333fcf1aa1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 03:58:27.929054 2639 raft_consensus.cc:348] T fe3f2fc004124efabd73cbef56710ac9 P 830c2976e7ef4eb4a96416333fcf1aa1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "553c7127b2b446729881acd33d89da44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35577 } } peers { permanent_uuid: "216d653228874c8bb95fa8cfdda70c04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33193 } } peers { permanent_uuid: "830c2976e7ef4eb4a96416333fcf1aa1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44451 } }
I20260926 03:58:27.929177 2639 raft_consensus.cc:374] T fe3f2fc004124efabd73cbef56710ac9 P 830c2976e7ef4eb4a96416333fcf1aa1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:58:27.929198 2639 raft_consensus.cc:729] T fe3f2fc004124efabd73cbef56710ac9 P 830c2976e7ef4eb4a96416333fcf1aa1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 830c2976e7ef4eb4a96416333fcf1aa1, State: Initialized, Role: FOLLOWER
I20260926 03:58:27.929324 2639 consensus_queue.cc:260] T fe3f2fc004124efabd73cbef56710ac9 P 830c2976e7ef4eb4a96416333fcf1aa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "553c7127b2b446729881acd33d89da44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35577 } } peers { permanent_uuid: "216d653228874c8bb95fa8cfdda70c04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33193 } } peers { permanent_uuid: "830c2976e7ef4eb4a96416333fcf1aa1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44451 } }
I20260926 03:58:27.929584 2474 heartbeater.cc:503] Master 127.0.98.62:42475 was elected leader, sending a full tablet report...
I20260926 03:58:27.929769 2639 ts_tablet_manager.cc:1428] T fe3f2fc004124efabd73cbef56710ac9 P 830c2976e7ef4eb4a96416333fcf1aa1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 03:58:27.942765 2640 raft_consensus.cc:482] T fe3f2fc004124efabd73cbef56710ac9 P 216d653228874c8bb95fa8cfdda70c04 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:58:27.942917 2640 raft_consensus.cc:504] T fe3f2fc004124efabd73cbef56710ac9 P 216d653228874c8bb95fa8cfdda70c04 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "553c7127b2b446729881acd33d89da44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35577 } } peers { permanent_uuid: "216d653228874c8bb95fa8cfdda70c04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33193 } } peers { permanent_uuid: "830c2976e7ef4eb4a96416333fcf1aa1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44451 } }
I20260926 03:58:27.943310 2640 leader_election.cc:288] T fe3f2fc004124efabd73cbef56710ac9 P 216d653228874c8bb95fa8cfdda70c04 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 553c7127b2b446729881acd33d89da44 (127.0.98.3:35577), 830c2976e7ef4eb4a96416333fcf1aa1 (127.0.98.2:44451)
I20260926 03:58:27.947126 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:27.943581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58973 (local address 127.0.98.3:35577)
0926 03:58:27.943639 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:27.943642 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:27.943673 (+ 31us) server_negotiation.cc:407] Connection header received
0926 03:58:27.943701 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:27.943701 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:27.943713 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:27.943765 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:58:27.943976 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:27.944784 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:27.946218 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:27.946764 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:27.946842 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:27.946852 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:27.947028 (+ 176us) server_negotiation.cc:299] Negotiation successful
0926 03:58:27.947066 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 03:58:27.947125 2227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:27.943510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35577 (local address 127.0.98.1:58973)
0926 03:58:27.943563 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:27.943576 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:27.943619 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:27.943820 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:27.943821 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:27.943823 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:58:27.943896 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:27.943898 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:27.944852 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:27.944853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:27.946130 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:27.946132 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:27.946868 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:27.946869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:27.946951 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:27.946956 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 03:58:27.947007 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 03:58:27.947053 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":64}
I20260926 03:58:27.947211 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:27.943703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56459 (local address 127.0.98.2:44451)
0926 03:58:27.943750 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:27.943752 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:27.943844 (+ 92us) server_negotiation.cc:407] Connection header received
0926 03:58:27.943954 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:27.943955 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:27.943965 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:27.944022 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:58:27.944140 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:27.944829 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:27.946218 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:27.946790 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:27.946845 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:27.946853 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:27.947095 (+ 242us) server_negotiation.cc:299] Negotiation successful
0926 03:58:27.947141 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 03:58:27.947436 2643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:27.943627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44451 (local address 127.0.98.1:56459)
0926 03:58:27.943793 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:27.943804 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:27.943855 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:27.944033 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:27.944033 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:27.944035 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:58:27.944079 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:27.944080 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:27.944919 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:27.944920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:27.946130 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:27.946132 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:27.946890 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:27.946891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:27.946999 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:27.947003 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 03:58:27.947298 (+ 295us) client_negotiation.cc:240] Negotiation successful
0926 03:58:27.947353 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":3,"thread_start_us":97,"threads_started":1}
I20260926 03:58:27.947448 2568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe3f2fc004124efabd73cbef56710ac9" candidate_uuid: "216d653228874c8bb95fa8cfdda70c04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "553c7127b2b446729881acd33d89da44" is_pre_election: true
I20260926 03:58:27.947605 2568 raft_consensus.cc:2454] T fe3f2fc004124efabd73cbef56710ac9 P 553c7127b2b446729881acd33d89da44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 216d653228874c8bb95fa8cfdda70c04 in term 0.
I20260926 03:58:27.947722 2428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe3f2fc004124efabd73cbef56710ac9" candidate_uuid: "216d653228874c8bb95fa8cfdda70c04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "830c2976e7ef4eb4a96416333fcf1aa1" is_pre_election: true
I20260926 03:58:27.947860 2428 raft_consensus.cc:2454] T fe3f2fc004124efabd73cbef56710ac9 P 830c2976e7ef4eb4a96416333fcf1aa1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 216d653228874c8bb95fa8cfdda70c04 in term 0.
I20260926 03:58:27.947844 2220 leader_election.cc:302] T fe3f2fc004124efabd73cbef56710ac9 P 216d653228874c8bb95fa8cfdda70c04 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 216d653228874c8bb95fa8cfdda70c04, 553c7127b2b446729881acd33d89da44; no voters:
I20260926 03:58:27.947973 2640 raft_consensus.cc:2781] T fe3f2fc004124efabd73cbef56710ac9 P 216d653228874c8bb95fa8cfdda70c04 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:58:27.948028 2640 raft_consensus.cc:482] T fe3f2fc004124efabd73cbef56710ac9 P 216d653228874c8bb95fa8cfdda70c04 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:58:27.948053 2640 raft_consensus.cc:3037] T fe3f2fc004124efabd73cbef56710ac9 P 216d653228874c8bb95fa8cfdda70c04 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:58:27.948716 2640 raft_consensus.cc:504] T fe3f2fc004124efabd73cbef56710ac9 P 216d653228874c8bb95fa8cfdda70c04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "553c7127b2b446729881acd33d89da44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35577 } } peers { permanent_uuid: "216d653228874c8bb95fa8cfdda70c04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33193 } } peers { permanent_uuid: "830c2976e7ef4eb4a96416333fcf1aa1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44451 } }
I20260926 03:58:27.948848 2640 leader_election.cc:288] T fe3f2fc004124efabd73cbef56710ac9 P 216d653228874c8bb95fa8cfdda70c04 [CANDIDATE]: Term 1 election: Requested vote from peers 553c7127b2b446729881acd33d89da44 (127.0.98.3:35577), 830c2976e7ef4eb4a96416333fcf1aa1 (127.0.98.2:44451)
I20260926 03:58:27.949021 2568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe3f2fc004124efabd73cbef56710ac9" candidate_uuid: "216d653228874c8bb95fa8cfdda70c04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "553c7127b2b446729881acd33d89da44"
I20260926 03:58:27.949008 2428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe3f2fc004124efabd73cbef56710ac9" candidate_uuid: "216d653228874c8bb95fa8cfdda70c04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "830c2976e7ef4eb4a96416333fcf1aa1"
I20260926 03:58:27.949107 2568 raft_consensus.cc:3037] T fe3f2fc004124efabd73cbef56710ac9 P 553c7127b2b446729881acd33d89da44 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:58:27.949105 2428 raft_consensus.cc:3037] T fe3f2fc004124efabd73cbef56710ac9 P 830c2976e7ef4eb4a96416333fcf1aa1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:58:27.949755 2428 raft_consensus.cc:2454] T fe3f2fc004124efabd73cbef56710ac9 P 830c2976e7ef4eb4a96416333fcf1aa1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 216d653228874c8bb95fa8cfdda70c04 in term 1.
I20260926 03:58:27.949808 2568 raft_consensus.cc:2454] T fe3f2fc004124efabd73cbef56710ac9 P 553c7127b2b446729881acd33d89da44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 216d653228874c8bb95fa8cfdda70c04 in term 1.
I20260926 03:58:27.949946 2220 leader_election.cc:302] T fe3f2fc004124efabd73cbef56710ac9 P 216d653228874c8bb95fa8cfdda70c04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 216d653228874c8bb95fa8cfdda70c04, 553c7127b2b446729881acd33d89da44; no voters:
I20260926 03:58:27.950054 2640 raft_consensus.cc:2781] T fe3f2fc004124efabd73cbef56710ac9 P 216d653228874c8bb95fa8cfdda70c04 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:58:27.950206 2640 raft_consensus.cc:686] T fe3f2fc004124efabd73cbef56710ac9 P 216d653228874c8bb95fa8cfdda70c04 [term 1 LEADER]: Becoming Leader. State: Replica: 216d653228874c8bb95fa8cfdda70c04, State: Running, Role: LEADER
I20260926 03:58:27.950304 2640 consensus_queue.cc:237] T fe3f2fc004124efabd73cbef56710ac9 P 216d653228874c8bb95fa8cfdda70c04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "553c7127b2b446729881acd33d89da44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35577 } } peers { permanent_uuid: "216d653228874c8bb95fa8cfdda70c04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33193 } } peers { permanent_uuid: "830c2976e7ef4eb4a96416333fcf1aa1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44451 } }
I20260926 03:58:27.951081 2130 catalog_manager.cc:5495] T fe3f2fc004124efabd73cbef56710ac9 P 216d653228874c8bb95fa8cfdda70c04 reported cstate change: term changed from 0 to 1, leader changed from <none> to 216d653228874c8bb95fa8cfdda70c04 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "216d653228874c8bb95fa8cfdda70c04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "553c7127b2b446729881acd33d89da44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "216d653228874c8bb95fa8cfdda70c04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "830c2976e7ef4eb4a96416333fcf1aa1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44451 } health_report { overall_health: UNKNOWN } } }
W20260926 03:58:27.976835 2475 tablet.cc:2369] T fe3f2fc004124efabd73cbef56710ac9 P 830c2976e7ef4eb4a96416333fcf1aa1: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 03:58:27 dist-test-slave-nhpw krb5kdc[2086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395107, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 03:58:27.984627 2199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:27.977396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43372 (local address 127.0.98.62:42475)
0926 03:58:27.977470 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:27.977473 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:27.977608 (+ 135us) server_negotiation.cc:407] Connection header received
0926 03:58:27.977688 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:27.977688 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:27.977699 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:27.977771 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:27.978234 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:27.978719 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:27.979424 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:27.979599 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:27.979687 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:27.981978 (+ 2291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:27.981980 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:27.981980 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:27.981987 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:27.983401 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:27.983793 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:27.983794 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:27.983794 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:27.983852 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:27.984081 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:27.984081 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:27.984082 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:27.984192 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:27.984264 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:27.984426 (+ 162us) server_negotiation.cc:299] Negotiation successful
0926 03:58:27.984521 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":34}
I20260926 03:58:27.984716 2651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:27.977329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42475 (local address 127.0.0.1:43372)
0926 03:58:27.977535 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:27.977545 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:27.977628 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:27.977775 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:27.977775 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:27.978104 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:27.978147 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:27.978149 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:27.978802 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:27.978803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:27.979329 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:27.979330 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:27.979699 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:27.979700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:27.979756 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:27.980310 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:27.980316 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:27.981875 (+ 1559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:58:27.983503 (+ 1628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:27.983510 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:27.983512 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:27.983703 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:27.983942 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:27.983943 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:27.983943 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:27.983999 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:27.984267 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:58:27.984274 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:58:27.984318 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 03:58:27.984403 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 03:58:27.984573 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":77,"thread_start_us":122,"threads_started":1}
I20260926 03:58:27.986873 2130 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43372:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 03:58:27.987087 2130 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:58:27.988584 2548 tablet_service.cc:1467] Processing CreateTablet for tablet ef7665a3754340dc805a94fa996e8a8f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1509ac325e3749889699290cdd79fb85]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 03:58:27.988698 2266 tablet_service.cc:1467] Processing CreateTablet for tablet ef7665a3754340dc805a94fa996e8a8f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1509ac325e3749889699290cdd79fb85]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 03:58:27.988797 2548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef7665a3754340dc805a94fa996e8a8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:58:27.988864 2266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef7665a3754340dc805a94fa996e8a8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:58:27.990073 2634 tablet_bootstrap.cc:492] T ef7665a3754340dc805a94fa996e8a8f P 553c7127b2b446729881acd33d89da44: Bootstrap starting.
I20260926 03:58:27.990073 2635 tablet_bootstrap.cc:492] T ef7665a3754340dc805a94fa996e8a8f P 216d653228874c8bb95fa8cfdda70c04: Bootstrap starting.
I20260926 03:58:27.990185 2408 tablet_service.cc:1467] Processing CreateTablet for tablet ef7665a3754340dc805a94fa996e8a8f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=1509ac325e3749889699290cdd79fb85]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 03:58:27.990319 2408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef7665a3754340dc805a94fa996e8a8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:58:27.990541 2635 tablet_bootstrap.cc:654] T ef7665a3754340dc805a94fa996e8a8f P 216d653228874c8bb95fa8cfdda70c04: Neither blocks nor log segments found. Creating new log.
I20260926 03:58:27.990618 2634 tablet_bootstrap.cc:654] T ef7665a3754340dc805a94fa996e8a8f P 553c7127b2b446729881acd33d89da44: Neither blocks nor log segments found. Creating new log.
I20260926 03:58:27.991247 2635 tablet_bootstrap.cc:492] T ef7665a3754340dc805a94fa996e8a8f P 216d653228874c8bb95fa8cfdda70c04: No bootstrap required, opened a new log
I20260926 03:58:27.991292 2635 ts_tablet_manager.cc:1397] T ef7665a3754340dc805a94fa996e8a8f P 216d653228874c8bb95fa8cfdda70c04: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 03:58:27.991444 2635 raft_consensus.cc:348] T ef7665a3754340dc805a94fa996e8a8f P 216d653228874c8bb95fa8cfdda70c04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "216d653228874c8bb95fa8cfdda70c04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33193 } } peers { permanent_uuid: "830c2976e7ef4eb4a96416333fcf1aa1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44451 } } peers { permanent_uuid: "553c7127b2b446729881acd33d89da44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35577 } }
I20260926 03:58:27.991523 2639 tablet_bootstrap.cc:492] T ef7665a3754340dc805a94fa996e8a8f P 830c2976e7ef4eb4a96416333fcf1aa1: Bootstrap starting.
I20260926 03:58:27.991559 2635 raft_consensus.cc:374] T ef7665a3754340dc805a94fa996e8a8f P 216d653228874c8bb95fa8cfdda70c04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:58:27.991581 2635 raft_consensus.cc:729] T ef7665a3754340dc805a94fa996e8a8f P 216d653228874c8bb95fa8cfdda70c04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 216d653228874c8bb95fa8cfdda70c04, State: Initialized, Role: FOLLOWER
I20260926 03:58:27.991621 2635 consensus_queue.cc:260] T ef7665a3754340dc805a94fa996e8a8f P 216d653228874c8bb95fa8cfdda70c04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "216d653228874c8bb95fa8cfdda70c04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33193 } } peers { permanent_uuid: "830c2976e7ef4eb4a96416333fcf1aa1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44451 } } peers { permanent_uuid: "553c7127b2b446729881acd33d89da44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35577 } }
I20260926 03:58:27.991735 2635 ts_tablet_manager.cc:1428] T ef7665a3754340dc805a94fa996e8a8f P 216d653228874c8bb95fa8cfdda70c04: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:27.991779 2640 tablet_replica.cc:439] T ef7665a3754340dc805a94fa996e8a8f P 216d653228874c8bb95fa8cfdda70c04: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "216d653228874c8bb95fa8cfdda70c04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33193 } } peers { permanent_uuid: "830c2976e7ef4eb4a96416333fcf1aa1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44451 } } peers { permanent_uuid: "553c7127b2b446729881acd33d89da44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35577 } } }
I20260926 03:58:27.991882 2640 tablet_replica.cc:442] T ef7665a3754340dc805a94fa996e8a8f P 216d653228874c8bb95fa8cfdda70c04: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:58:27.992185 2639 tablet_bootstrap.cc:654] T ef7665a3754340dc805a94fa996e8a8f P 830c2976e7ef4eb4a96416333fcf1aa1: Neither blocks nor log segments found. Creating new log.
I20260926 03:58:27.992682 2634 tablet_bootstrap.cc:492] T ef7665a3754340dc805a94fa996e8a8f P 553c7127b2b446729881acd33d89da44: No bootstrap required, opened a new log
I20260926 03:58:27.992734 2634 ts_tablet_manager.cc:1397] T ef7665a3754340dc805a94fa996e8a8f P 553c7127b2b446729881acd33d89da44: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 03:58:27.992905 2639 tablet_bootstrap.cc:492] T ef7665a3754340dc805a94fa996e8a8f P 830c2976e7ef4eb4a96416333fcf1aa1: No bootstrap required, opened a new log
I20260926 03:58:27.992874 2634 raft_consensus.cc:348] T ef7665a3754340dc805a94fa996e8a8f P 553c7127b2b446729881acd33d89da44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "216d653228874c8bb95fa8cfdda70c04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33193 } } peers { permanent_uuid: "830c2976e7ef4eb4a96416333fcf1aa1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44451 } } peers { permanent_uuid: "553c7127b2b446729881acd33d89da44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35577 } }
I20260926 03:58:27.992962 2639 ts_tablet_manager.cc:1397] T ef7665a3754340dc805a94fa996e8a8f P 830c2976e7ef4eb4a96416333fcf1aa1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 03:58:27.992967 2634 raft_consensus.cc:374] T ef7665a3754340dc805a94fa996e8a8f P 553c7127b2b446729881acd33d89da44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:58:27.992996 2634 raft_consensus.cc:729] T ef7665a3754340dc805a94fa996e8a8f P 553c7127b2b446729881acd33d89da44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 553c7127b2b446729881acd33d89da44, State: Initialized, Role: FOLLOWER
I20260926 03:58:27.993043 2634 consensus_queue.cc:260] T ef7665a3754340dc805a94fa996e8a8f P 553c7127b2b446729881acd33d89da44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "216d653228874c8bb95fa8cfdda70c04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33193 } } peers { permanent_uuid: "830c2976e7ef4eb4a96416333fcf1aa1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44451 } } peers { permanent_uuid: "553c7127b2b446729881acd33d89da44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35577 } }
I20260926 03:58:27.993134 2634 ts_tablet_manager.cc:1428] T ef7665a3754340dc805a94fa996e8a8f P 553c7127b2b446729881acd33d89da44: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:27.993112 2639 raft_consensus.cc:348] T ef7665a3754340dc805a94fa996e8a8f P 830c2976e7ef4eb4a96416333fcf1aa1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "216d653228874c8bb95fa8cfdda70c04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33193 } } peers { permanent_uuid: "830c2976e7ef4eb4a96416333fcf1aa1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44451 } } peers { permanent_uuid: "553c7127b2b446729881acd33d89da44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35577 } }
I20260926 03:58:27.993175 2639 raft_consensus.cc:374] T ef7665a3754340dc805a94fa996e8a8f P 830c2976e7ef4eb4a96416333fcf1aa1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:58:27.993202 2639 raft_consensus.cc:729] T ef7665a3754340dc805a94fa996e8a8f P 830c2976e7ef4eb4a96416333fcf1aa1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 830c2976e7ef4eb4a96416333fcf1aa1, State: Initialized, Role: FOLLOWER
I20260926 03:58:27.993187 2638 tablet_replica.cc:439] T ef7665a3754340dc805a94fa996e8a8f P 553c7127b2b446729881acd33d89da44: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "216d653228874c8bb95fa8cfdda70c04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33193 } } peers { permanent_uuid: "830c2976e7ef4eb4a96416333fcf1aa1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44451 } } peers { permanent_uuid: "553c7127b2b446729881acd33d89da44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35577 } } }
I20260926 03:58:27.993257 2639 consensus_queue.cc:260] T ef7665a3754340dc805a94fa996e8a8f P 830c2976e7ef4eb4a96416333fcf1aa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "216d653228874c8bb95fa8cfdda70c04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33193 } } peers { permanent_uuid: "830c2976e7ef4eb4a96416333fcf1aa1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44451 } } peers { permanent_uuid: "553c7127b2b446729881acd33d89da44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35577 } }
I20260926 03:58:27.993285 2638 tablet_replica.cc:442] T ef7665a3754340dc805a94fa996e8a8f P 553c7127b2b446729881acd33d89da44: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:58:27.993355 2639 ts_tablet_manager.cc:1428] T ef7665a3754340dc805a94fa996e8a8f P 830c2976e7ef4eb4a96416333fcf1aa1: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 03:58:27.993427 2642 tablet_replica.cc:439] T ef7665a3754340dc805a94fa996e8a8f P 830c2976e7ef4eb4a96416333fcf1aa1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "216d653228874c8bb95fa8cfdda70c04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33193 } } peers { permanent_uuid: "830c2976e7ef4eb4a96416333fcf1aa1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44451 } } peers { permanent_uuid: "553c7127b2b446729881acd33d89da44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35577 } } }
I20260926 03:58:27.993521 2642 tablet_replica.cc:442] T ef7665a3754340dc805a94fa996e8a8f P 830c2976e7ef4eb4a96416333fcf1aa1: This TxnStatusTablet replica's current role is: FOLLOWER
W20260926 03:58:28.076792 2335 tablet.cc:2369] T ef7665a3754340dc805a94fa996e8a8f P 216d653228874c8bb95fa8cfdda70c04: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:58:28.100708 2638 raft_consensus.cc:482] T ef7665a3754340dc805a94fa996e8a8f P 553c7127b2b446729881acd33d89da44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:58:28.100831 2638 raft_consensus.cc:504] T ef7665a3754340dc805a94fa996e8a8f P 553c7127b2b446729881acd33d89da44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "216d653228874c8bb95fa8cfdda70c04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33193 } } peers { permanent_uuid: "830c2976e7ef4eb4a96416333fcf1aa1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44451 } } peers { permanent_uuid: "553c7127b2b446729881acd33d89da44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35577 } }
I20260926 03:58:28.101220 2638 leader_election.cc:288] T ef7665a3754340dc805a94fa996e8a8f P 553c7127b2b446729881acd33d89da44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 216d653228874c8bb95fa8cfdda70c04 (127.0.98.1:33193), 830c2976e7ef4eb4a96416333fcf1aa1 (127.0.98.2:44451)
I20260926 03:58:28.103708 2640 raft_consensus.cc:482] T ef7665a3754340dc805a94fa996e8a8f P 216d653228874c8bb95fa8cfdda70c04 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:58:28.103798 2640 raft_consensus.cc:504] T ef7665a3754340dc805a94fa996e8a8f P 216d653228874c8bb95fa8cfdda70c04 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "216d653228874c8bb95fa8cfdda70c04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33193 } } peers { permanent_uuid: "830c2976e7ef4eb4a96416333fcf1aa1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44451 } } peers { permanent_uuid: "553c7127b2b446729881acd33d89da44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35577 } }
I20260926 03:58:28.103927 2640 leader_election.cc:288] T ef7665a3754340dc805a94fa996e8a8f P 216d653228874c8bb95fa8cfdda70c04 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 830c2976e7ef4eb4a96416333fcf1aa1 (127.0.98.2:44451), 553c7127b2b446729881acd33d89da44 (127.0.98.3:35577)
I20260926 03:58:28.104223 2428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef7665a3754340dc805a94fa996e8a8f" candidate_uuid: "216d653228874c8bb95fa8cfdda70c04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "830c2976e7ef4eb4a96416333fcf1aa1" is_pre_election: true
I20260926 03:58:28.104212 2568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef7665a3754340dc805a94fa996e8a8f" candidate_uuid: "216d653228874c8bb95fa8cfdda70c04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "553c7127b2b446729881acd33d89da44" is_pre_election: true
I20260926 03:58:28.104300 2568 raft_consensus.cc:2454] T ef7665a3754340dc805a94fa996e8a8f P 553c7127b2b446729881acd33d89da44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 216d653228874c8bb95fa8cfdda70c04 in term 0.
I20260926 03:58:28.104300 2428 raft_consensus.cc:2454] T ef7665a3754340dc805a94fa996e8a8f P 830c2976e7ef4eb4a96416333fcf1aa1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 216d653228874c8bb95fa8cfdda70c04 in term 0.
I20260926 03:58:28.104434 2507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:28.101441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44451 (local address 127.0.98.3:50349)
0926 03:58:28.101500 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:28.101513 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:28.101562 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:28.101732 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:28.101733 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:28.101735 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:58:28.101809 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:28.101810 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:28.102626 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:28.102627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:28.103381 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:28.103383 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:28.104146 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:28.104148 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:28.104235 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:28.104239 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 03:58:28.104292 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 03:58:28.104330 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 03:58:28.104437 2220 leader_election.cc:302] T ef7665a3754340dc805a94fa996e8a8f P 216d653228874c8bb95fa8cfdda70c04 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 216d653228874c8bb95fa8cfdda70c04, 830c2976e7ef4eb4a96416333fcf1aa1; no voters:
I20260926 03:58:28.104542 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:28.101497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45891 (local address 127.0.98.1:33193)
0926 03:58:28.101548 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:28.101550 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:28.101705 (+ 155us) server_negotiation.cc:407] Connection header received
0926 03:58:28.101767 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:28.101768 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:28.101779 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:28.101861 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:58:28.102157 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:28.102632 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:28.103549 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:28.104174 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:28.104235 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:28.104245 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:28.104424 (+ 179us) server_negotiation.cc:299] Negotiation successful
0926 03:58:28.104477 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 03:58:28.104678 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:28.101495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50349 (local address 127.0.98.2:44451)
0926 03:58:28.101546 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:28.101548 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:28.101559 (+ 11us) server_negotiation.cc:407] Connection header received
0926 03:58:28.101632 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:28.101632 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:28.101639 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:28.101728 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:58:28.101884 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:28.102507 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:28.103512 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:28.104052 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:28.104145 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:28.104157 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:28.104548 (+ 391us) server_negotiation.cc:299] Negotiation successful
0926 03:58:28.104592 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 03:58:28.104671 2640 raft_consensus.cc:2781] T ef7665a3754340dc805a94fa996e8a8f P 216d653228874c8bb95fa8cfdda70c04 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:58:28.104709 2640 raft_consensus.cc:482] T ef7665a3754340dc805a94fa996e8a8f P 216d653228874c8bb95fa8cfdda70c04 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:58:28.104733 2640 raft_consensus.cc:3037] T ef7665a3754340dc805a94fa996e8a8f P 216d653228874c8bb95fa8cfdda70c04 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:58:28.104727 2655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:28.101460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33193 (local address 127.0.98.3:45891)
0926 03:58:28.101621 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:28.101630 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:28.101711 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:28.101867 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:28.101868 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:28.101870 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:58:28.101939 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:28.101941 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:28.102718 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:28.102719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:28.103454 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:28.103456 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:28.104291 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:28.104292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:28.104347 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:28.104350 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 03:58:28.104601 (+ 251us) client_negotiation.cc:240] Negotiation successful
0926 03:58:28.104654 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":114,"threads_started":1}
I20260926 03:58:28.105012 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef7665a3754340dc805a94fa996e8a8f" candidate_uuid: "553c7127b2b446729881acd33d89da44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "216d653228874c8bb95fa8cfdda70c04" is_pre_election: true
I20260926 03:58:28.105446 2428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef7665a3754340dc805a94fa996e8a8f" candidate_uuid: "553c7127b2b446729881acd33d89da44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "830c2976e7ef4eb4a96416333fcf1aa1" is_pre_election: true
I20260926 03:58:28.105535 2428 raft_consensus.cc:2454] T ef7665a3754340dc805a94fa996e8a8f P 830c2976e7ef4eb4a96416333fcf1aa1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 553c7127b2b446729881acd33d89da44 in term 0.
I20260926 03:58:28.105567 2640 raft_consensus.cc:504] T ef7665a3754340dc805a94fa996e8a8f P 216d653228874c8bb95fa8cfdda70c04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "216d653228874c8bb95fa8cfdda70c04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33193 } } peers { permanent_uuid: "830c2976e7ef4eb4a96416333fcf1aa1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44451 } } peers { permanent_uuid: "553c7127b2b446729881acd33d89da44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35577 } }
I20260926 03:58:28.105698 2640 leader_election.cc:288] T ef7665a3754340dc805a94fa996e8a8f P 216d653228874c8bb95fa8cfdda70c04 [CANDIDATE]: Term 1 election: Requested vote from peers 830c2976e7ef4eb4a96416333fcf1aa1 (127.0.98.2:44451), 553c7127b2b446729881acd33d89da44 (127.0.98.3:35577)
I20260926 03:58:28.105695 2500 leader_election.cc:302] T ef7665a3754340dc805a94fa996e8a8f P 553c7127b2b446729881acd33d89da44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 553c7127b2b446729881acd33d89da44, 830c2976e7ef4eb4a96416333fcf1aa1; no voters:
I20260926 03:58:28.105782 2288 raft_consensus.cc:2379] T ef7665a3754340dc805a94fa996e8a8f P 216d653228874c8bb95fa8cfdda70c04 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 553c7127b2b446729881acd33d89da44 in current term 1: Already voted for candidate 216d653228874c8bb95fa8cfdda70c04 in this term.
I20260926 03:58:28.105826 2638 raft_consensus.cc:2781] T ef7665a3754340dc805a94fa996e8a8f P 553c7127b2b446729881acd33d89da44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:58:28.105862 2638 raft_consensus.cc:482] T ef7665a3754340dc805a94fa996e8a8f P 553c7127b2b446729881acd33d89da44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:58:28.105906 2638 raft_consensus.cc:3037] T ef7665a3754340dc805a94fa996e8a8f P 553c7127b2b446729881acd33d89da44 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:58:28.105868 2428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef7665a3754340dc805a94fa996e8a8f" candidate_uuid: "216d653228874c8bb95fa8cfdda70c04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "830c2976e7ef4eb4a96416333fcf1aa1"
I20260926 03:58:28.105948 2428 raft_consensus.cc:3037] T ef7665a3754340dc805a94fa996e8a8f P 830c2976e7ef4eb4a96416333fcf1aa1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:58:28.106333 2568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef7665a3754340dc805a94fa996e8a8f" candidate_uuid: "216d653228874c8bb95fa8cfdda70c04" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "553c7127b2b446729881acd33d89da44"
I20260926 03:58:28.106717 2638 raft_consensus.cc:504] T ef7665a3754340dc805a94fa996e8a8f P 553c7127b2b446729881acd33d89da44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "216d653228874c8bb95fa8cfdda70c04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33193 } } peers { permanent_uuid: "830c2976e7ef4eb4a96416333fcf1aa1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44451 } } peers { permanent_uuid: "553c7127b2b446729881acd33d89da44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35577 } }
I20260926 03:58:28.106765 2428 raft_consensus.cc:2454] T ef7665a3754340dc805a94fa996e8a8f P 830c2976e7ef4eb4a96416333fcf1aa1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 216d653228874c8bb95fa8cfdda70c04 in term 1.
I20260926 03:58:28.106873 2638 leader_election.cc:288] T ef7665a3754340dc805a94fa996e8a8f P 553c7127b2b446729881acd33d89da44 [CANDIDATE]: Term 1 election: Requested vote from peers 216d653228874c8bb95fa8cfdda70c04 (127.0.98.1:33193), 830c2976e7ef4eb4a96416333fcf1aa1 (127.0.98.2:44451)
I20260926 03:58:28.106947 2568 raft_consensus.cc:2379] T ef7665a3754340dc805a94fa996e8a8f P 553c7127b2b446729881acd33d89da44 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 216d653228874c8bb95fa8cfdda70c04 in current term 1: Already voted for candidate 553c7127b2b446729881acd33d89da44 in this term.
I20260926 03:58:28.106982 2428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef7665a3754340dc805a94fa996e8a8f" candidate_uuid: "553c7127b2b446729881acd33d89da44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "830c2976e7ef4eb4a96416333fcf1aa1"
I20260926 03:58:28.107020 2220 leader_election.cc:302] T ef7665a3754340dc805a94fa996e8a8f P 216d653228874c8bb95fa8cfdda70c04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 216d653228874c8bb95fa8cfdda70c04, 830c2976e7ef4eb4a96416333fcf1aa1; no voters:
I20260926 03:58:28.107074 2428 raft_consensus.cc:2379] T ef7665a3754340dc805a94fa996e8a8f P 830c2976e7ef4eb4a96416333fcf1aa1 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 553c7127b2b446729881acd33d89da44 in current term 1: Already voted for candidate 216d653228874c8bb95fa8cfdda70c04 in this term.
I20260926 03:58:28.107090 2640 raft_consensus.cc:2781] T ef7665a3754340dc805a94fa996e8a8f P 216d653228874c8bb95fa8cfdda70c04 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:58:28.107059 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef7665a3754340dc805a94fa996e8a8f" candidate_uuid: "553c7127b2b446729881acd33d89da44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "216d653228874c8bb95fa8cfdda70c04"
I20260926 03:58:28.107134 2640 raft_consensus.cc:686] T ef7665a3754340dc805a94fa996e8a8f P 216d653228874c8bb95fa8cfdda70c04 [term 1 LEADER]: Becoming Leader. State: Replica: 216d653228874c8bb95fa8cfdda70c04, State: Running, Role: LEADER
I20260926 03:58:28.107172 2640 consensus_queue.cc:237] T ef7665a3754340dc805a94fa996e8a8f P 216d653228874c8bb95fa8cfdda70c04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "216d653228874c8bb95fa8cfdda70c04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33193 } } peers { permanent_uuid: "830c2976e7ef4eb4a96416333fcf1aa1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44451 } } peers { permanent_uuid: "553c7127b2b446729881acd33d89da44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35577 } }
I20260926 03:58:28.107450 2644 tablet_replica.cc:439] T ef7665a3754340dc805a94fa996e8a8f P 216d653228874c8bb95fa8cfdda70c04: TxnStatusTablet state changed. Reason: New leader 216d653228874c8bb95fa8cfdda70c04. Latest consensus state: current_term: 1 leader_uuid: "216d653228874c8bb95fa8cfdda70c04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "216d653228874c8bb95fa8cfdda70c04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33193 } } peers { permanent_uuid: "830c2976e7ef4eb4a96416333fcf1aa1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44451 } } peers { permanent_uuid: "553c7127b2b446729881acd33d89da44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35577 } } }
I20260926 03:58:28.107532 2644 tablet_replica.cc:442] T ef7665a3754340dc805a94fa996e8a8f P 216d653228874c8bb95fa8cfdda70c04: This TxnStatusTablet replica's current role is: LEADER
I20260926 03:58:28.107534 2499 leader_election.cc:302] T ef7665a3754340dc805a94fa996e8a8f P 553c7127b2b446729881acd33d89da44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 553c7127b2b446729881acd33d89da44; no voters: 216d653228874c8bb95fa8cfdda70c04, 830c2976e7ef4eb4a96416333fcf1aa1
I20260926 03:58:28.107692 2657 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 03:58:28.107738 2638 raft_consensus.cc:2726] T ef7665a3754340dc805a94fa996e8a8f P 553c7127b2b446729881acd33d89da44 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260926 03:58:28.107863 2657 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 03:58:28.107947 2130 catalog_manager.cc:5495] T ef7665a3754340dc805a94fa996e8a8f P 216d653228874c8bb95fa8cfdda70c04 reported cstate change: term changed from 0 to 1, leader changed from <none> to 216d653228874c8bb95fa8cfdda70c04 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "216d653228874c8bb95fa8cfdda70c04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "216d653228874c8bb95fa8cfdda70c04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "830c2976e7ef4eb4a96416333fcf1aa1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "553c7127b2b446729881acd33d89da44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35577 } health_report { overall_health: UNKNOWN } } }
W20260926 03:58:28.128321 2615 tablet.cc:2369] T ef7665a3754340dc805a94fa996e8a8f P 553c7127b2b446729881acd33d89da44: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:58:28.139299 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:28.136829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42874 (local address 127.0.98.1:33193)
0926 03:58:28.136895 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:28.136897 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:28.136907 (+ 10us) server_negotiation.cc:407] Connection header received
0926 03:58:28.137024 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:28.137025 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:28.137045 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:28.137106 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:58:28.137238 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:28.137735 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:28.138502 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:28.138709 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:28.138787 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:28.138966 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:58:28.139071 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:58:28.139162 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:28.139202 (+ 40us) server_negotiation.cc:299] Negotiation successful
0926 03:58:28.139223 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 03:58:28.139314 2651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:28.136756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33193 (local address 127.0.0.1:42874)
0926 03:58:28.136841 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:28.136872 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:28.136925 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:28.137109 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:28.137110 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:28.137111 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 03:58:28.137161 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:28.137164 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:28.137826 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:28.137827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:28.138401 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:28.138403 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:28.138787 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:28.138788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:28.138859 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:28.138871 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 03:58:28.139142 (+ 271us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 03:58:28.139151 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 03:58:28.139198 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 03:58:28.139247 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 03:58:28.141127 2428 raft_consensus.cc:1261] T ef7665a3754340dc805a94fa996e8a8f P 830c2976e7ef4eb4a96416333fcf1aa1 [term 1 FOLLOWER]: Refusing update from remote peer 216d653228874c8bb95fa8cfdda70c04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:58:28.141141 2568 raft_consensus.cc:1261] T ef7665a3754340dc805a94fa996e8a8f P 553c7127b2b446729881acd33d89da44 [term 1 FOLLOWER]: Refusing update from remote peer 216d653228874c8bb95fa8cfdda70c04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:58:28.141371 2644 consensus_queue.cc:1035] T ef7665a3754340dc805a94fa996e8a8f P 216d653228874c8bb95fa8cfdda70c04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "553c7127b2b446729881acd33d89da44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:58:28.141518 2640 consensus_queue.cc:1035] T ef7665a3754340dc805a94fa996e8a8f P 216d653228874c8bb95fa8cfdda70c04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "830c2976e7ef4eb4a96416333fcf1aa1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:58:28.142642 2638 tablet_replica.cc:439] T ef7665a3754340dc805a94fa996e8a8f P 553c7127b2b446729881acd33d89da44: TxnStatusTablet state changed. Reason: New leader 216d653228874c8bb95fa8cfdda70c04. Latest consensus state: current_term: 1 leader_uuid: "216d653228874c8bb95fa8cfdda70c04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "216d653228874c8bb95fa8cfdda70c04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33193 } } peers { permanent_uuid: "830c2976e7ef4eb4a96416333fcf1aa1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44451 } } peers { permanent_uuid: "553c7127b2b446729881acd33d89da44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35577 } } }
I20260926 03:58:28.142745 2638 tablet_replica.cc:442] T ef7665a3754340dc805a94fa996e8a8f P 553c7127b2b446729881acd33d89da44: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:58:28.142831 2642 tablet_replica.cc:439] T ef7665a3754340dc805a94fa996e8a8f P 830c2976e7ef4eb4a96416333fcf1aa1: TxnStatusTablet state changed. Reason: New leader 216d653228874c8bb95fa8cfdda70c04. Latest consensus state: current_term: 1 leader_uuid: "216d653228874c8bb95fa8cfdda70c04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "216d653228874c8bb95fa8cfdda70c04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33193 } } peers { permanent_uuid: "830c2976e7ef4eb4a96416333fcf1aa1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44451 } } peers { permanent_uuid: "553c7127b2b446729881acd33d89da44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35577 } } }
I20260926 03:58:28.142970 2642 tablet_replica.cc:442] T ef7665a3754340dc805a94fa996e8a8f P 830c2976e7ef4eb4a96416333fcf1aa1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:58:28.143863 2640 tablet_replica.cc:439] T ef7665a3754340dc805a94fa996e8a8f P 216d653228874c8bb95fa8cfdda70c04: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "216d653228874c8bb95fa8cfdda70c04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "216d653228874c8bb95fa8cfdda70c04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33193 } } peers { permanent_uuid: "830c2976e7ef4eb4a96416333fcf1aa1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44451 } } peers { permanent_uuid: "553c7127b2b446729881acd33d89da44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35577 } } }
I20260926 03:58:28.143949 2640 tablet_replica.cc:442] T ef7665a3754340dc805a94fa996e8a8f P 216d653228874c8bb95fa8cfdda70c04: This TxnStatusTablet replica's current role is: LEADER
I20260926 03:58:28.144316 2644 tablet_replica.cc:439] T ef7665a3754340dc805a94fa996e8a8f P 216d653228874c8bb95fa8cfdda70c04: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "216d653228874c8bb95fa8cfdda70c04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "216d653228874c8bb95fa8cfdda70c04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33193 } } peers { permanent_uuid: "830c2976e7ef4eb4a96416333fcf1aa1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44451 } } peers { permanent_uuid: "553c7127b2b446729881acd33d89da44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35577 } } }
I20260926 03:58:28.144393 2644 tablet_replica.cc:442] T ef7665a3754340dc805a94fa996e8a8f P 216d653228874c8bb95fa8cfdda70c04: This TxnStatusTablet replica's current role is: LEADER
I20260926 03:58:28.144886 2638 tablet_replica.cc:439] T ef7665a3754340dc805a94fa996e8a8f P 553c7127b2b446729881acd33d89da44: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "216d653228874c8bb95fa8cfdda70c04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "216d653228874c8bb95fa8cfdda70c04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33193 } } peers { permanent_uuid: "830c2976e7ef4eb4a96416333fcf1aa1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44451 } } peers { permanent_uuid: "553c7127b2b446729881acd33d89da44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35577 } } }
I20260926 03:58:28.144970 2638 tablet_replica.cc:442] T ef7665a3754340dc805a94fa996e8a8f P 553c7127b2b446729881acd33d89da44: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:58:28.145458 2642 tablet_replica.cc:439] T ef7665a3754340dc805a94fa996e8a8f P 830c2976e7ef4eb4a96416333fcf1aa1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "216d653228874c8bb95fa8cfdda70c04" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "216d653228874c8bb95fa8cfdda70c04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33193 } } peers { permanent_uuid: "830c2976e7ef4eb4a96416333fcf1aa1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44451 } } peers { permanent_uuid: "553c7127b2b446729881acd33d89da44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35577 } } }
I20260926 03:58:28.145539 2642 tablet_replica.cc:442] T ef7665a3754340dc805a94fa996e8a8f P 830c2976e7ef4eb4a96416333fcf1aa1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:58:28.145577 2662 mvcc.cc:205] Tried to move back new op lower bound from 7333458362943651840 to 7333458362807300096. Current Snapshot: MvccSnapshot[applied={T|T < 7333458362943651840}]
I20260926 03:58:28.148866 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:28.146615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42888 (local address 127.0.98.1:33193)
0926 03:58:28.146656 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:28.146658 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:28.146667 (+ 9us) server_negotiation.cc:407] Connection header received
0926 03:58:28.146728 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:28.146729 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:28.146740 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:28.146794 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:58:28.146916 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:28.147370 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:28.148166 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:28.148366 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:28.148454 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:28.148613 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:58:28.148668 (+ 55us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:58:28.148713 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:28.148778 (+ 65us) server_negotiation.cc:299] Negotiation successful
0926 03:58:28.148794 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 03:58:28.153762 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:28.151361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58527 (local address 127.0.98.1:33193)
0926 03:58:28.151402 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:28.151404 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:28.151496 (+ 92us) server_negotiation.cc:407] Connection header received
0926 03:58:28.151554 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:28.151555 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:28.151564 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:28.151617 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:58:28.151760 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:28.152188 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:28.153043 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:28.153217 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:28.153297 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:28.153484 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:58:28.153519 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:58:28.153575 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:28.153642 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 03:58:28.153674 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":13,"server-negotiator.queue_time_us":22}
I20260926 03:58:28.153781 2671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:28.151284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33193 (local address 127.0.98.1:58527)
0926 03:58:28.151418 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:28.151437 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:28.151497 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:28.151620 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:28.151620 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:28.151621 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 03:58:28.151691 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:28.151693 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:28.152276 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:28.152277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:28.152953 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:28.152955 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:28.153298 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:28.153299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:28.153367 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:28.153378 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 03:58:28.153588 (+ 210us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 03:58:28.153590 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 03:58:28.153638 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 03:58:28.153692 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"mutex_wait_us":2,"thread_start_us":79,"threads_started":1}
I20260926 03:58:28.155737 2567 raft_consensus.cc:1261] T fe3f2fc004124efabd73cbef56710ac9 P 553c7127b2b446729881acd33d89da44 [term 1 FOLLOWER]: Refusing update from remote peer 216d653228874c8bb95fa8cfdda70c04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:58:28.155898 2428 raft_consensus.cc:1261] T fe3f2fc004124efabd73cbef56710ac9 P 830c2976e7ef4eb4a96416333fcf1aa1 [term 1 FOLLOWER]: Refusing update from remote peer 216d653228874c8bb95fa8cfdda70c04: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:58:28.156009 2640 consensus_queue.cc:1035] T fe3f2fc004124efabd73cbef56710ac9 P 216d653228874c8bb95fa8cfdda70c04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "553c7127b2b446729881acd33d89da44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:58:28.156273 2644 consensus_queue.cc:1035] T fe3f2fc004124efabd73cbef56710ac9 P 216d653228874c8bb95fa8cfdda70c04 [LEADER]: Connected to new peer: Peer: permanent_uuid: "830c2976e7ef4eb4a96416333fcf1aa1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:58:28.157510 2660 mvcc.cc:205] Tried to move back new op lower bound from 7333458363004284928 to 7333458362165035008. Current Snapshot: MvccSnapshot[applied={T|T < 7333458363004284928}]
I20260926 03:58:28.157639 2661 mvcc.cc:205] Tried to move back new op lower bound from 7333458363004284928 to 7333458362165035008. Current Snapshot: MvccSnapshot[applied={T|T < 7333458363004284928}]
I20260926 03:58:28.167798 2131 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43364:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:58:28.167894 2131 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43364:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:58:28.168573 2131 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d3414598d0e6464bb4b6f634282fb92a: Sending DeleteTablet for 3 replicas of tablet fe3f2fc004124efabd73cbef56710ac9
I20260926 03:58:28.168915 2268 tablet_service.cc:1514] Processing DeleteTablet for tablet fe3f2fc004124efabd73cbef56710ac9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:58:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42872
I20260926 03:58:28.169009 2408 tablet_service.cc:1514] Processing DeleteTablet for tablet fe3f2fc004124efabd73cbef56710ac9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:58:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55982
I20260926 03:58:28.169154 2676 tablet_replica.cc:330] T fe3f2fc004124efabd73cbef56710ac9 P 216d653228874c8bb95fa8cfdda70c04: stopping tablet replica
I20260926 03:58:28.169142 2548 tablet_service.cc:1514] Processing DeleteTablet for tablet fe3f2fc004124efabd73cbef56710ac9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:58:28 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59398
I20260926 03:58:28.169243 2676 raft_consensus.cc:2229] T fe3f2fc004124efabd73cbef56710ac9 P 216d653228874c8bb95fa8cfdda70c04 [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:58:28.169327 2676 raft_consensus.cc:2258] T fe3f2fc004124efabd73cbef56710ac9 P 216d653228874c8bb95fa8cfdda70c04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:58:28.169648 2678 tablet_replica.cc:330] T fe3f2fc004124efabd73cbef56710ac9 P 553c7127b2b446729881acd33d89da44: stopping tablet replica
I20260926 03:58:28.169693 2676 ts_tablet_manager.cc:1905] T fe3f2fc004124efabd73cbef56710ac9 P 216d653228874c8bb95fa8cfdda70c04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:58:28.169744 2678 raft_consensus.cc:2229] T fe3f2fc004124efabd73cbef56710ac9 P 553c7127b2b446729881acd33d89da44 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:58:28.169790 2678 raft_consensus.cc:2258] T fe3f2fc004124efabd73cbef56710ac9 P 553c7127b2b446729881acd33d89da44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:58:28.169735 2131 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43364:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:58:28.169865 2131 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:58:28.170029 2677 tablet_replica.cc:330] T fe3f2fc004124efabd73cbef56710ac9 P 830c2976e7ef4eb4a96416333fcf1aa1: stopping tablet replica
I20260926 03:58:28.170152 2677 raft_consensus.cc:2229] T fe3f2fc004124efabd73cbef56710ac9 P 830c2976e7ef4eb4a96416333fcf1aa1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:58:28.170212 2677 raft_consensus.cc:2258] T fe3f2fc004124efabd73cbef56710ac9 P 830c2976e7ef4eb4a96416333fcf1aa1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:58:28.170644 2677 ts_tablet_manager.cc:1905] T fe3f2fc004124efabd73cbef56710ac9 P 830c2976e7ef4eb4a96416333fcf1aa1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:58:28.171168 2268 tablet_service.cc:1467] Processing CreateTablet for tablet c328a979844741a6b1e7daee9d1cfaca (DEFAULT_TABLE table=test-table [id=e9ff458e976e490c9c6a4dcc0436e05f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:58:28.171290 2268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c328a979844741a6b1e7daee9d1cfaca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:58:28.171623 2548 tablet_service.cc:1467] Processing CreateTablet for tablet c328a979844741a6b1e7daee9d1cfaca (DEFAULT_TABLE table=test-table [id=e9ff458e976e490c9c6a4dcc0436e05f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:58:28.171729 2548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c328a979844741a6b1e7daee9d1cfaca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:58:28.172305 2678 ts_tablet_manager.cc:1905] T fe3f2fc004124efabd73cbef56710ac9 P 553c7127b2b446729881acd33d89da44: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:58:28.172370 2677 ts_tablet_manager.cc:1918] T fe3f2fc004124efabd73cbef56710ac9 P 830c2976e7ef4eb4a96416333fcf1aa1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 03:58:28.172461 2677 log.cc:1196] T fe3f2fc004124efabd73cbef56710ac9 P 830c2976e7ef4eb4a96416333fcf1aa1: Deleting WAL directory at /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/minicluster-data/ts-1/wal/wals/fe3f2fc004124efabd73cbef56710ac9
I20260926 03:58:28.172751 2677 ts_tablet_manager.cc:1939] T fe3f2fc004124efabd73cbef56710ac9 P 830c2976e7ef4eb4a96416333fcf1aa1: Deleting consensus metadata
I20260926 03:58:28.173133 2116 catalog_manager.cc:4841] TS 830c2976e7ef4eb4a96416333fcf1aa1 (127.0.98.2:44451): tablet fe3f2fc004124efabd73cbef56710ac9 (table test-table [id=61eebd3e319541018adc5d24d7865279]) successfully deleted
I20260926 03:58:28.173161 2634 tablet_bootstrap.cc:492] T c328a979844741a6b1e7daee9d1cfaca P 553c7127b2b446729881acd33d89da44: Bootstrap starting.
I20260926 03:58:28.173595 2634 tablet_bootstrap.cc:654] T c328a979844741a6b1e7daee9d1cfaca P 553c7127b2b446729881acd33d89da44: Neither blocks nor log segments found. Creating new log.
I20260926 03:58:28.174057 2408 tablet_service.cc:1467] Processing CreateTablet for tablet c328a979844741a6b1e7daee9d1cfaca (DEFAULT_TABLE table=test-table [id=e9ff458e976e490c9c6a4dcc0436e05f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:58:28.174151 2408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c328a979844741a6b1e7daee9d1cfaca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:58:28.174151 2678 ts_tablet_manager.cc:1918] T fe3f2fc004124efabd73cbef56710ac9 P 553c7127b2b446729881acd33d89da44: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 03:58:28.174207 2678 log.cc:1196] T fe3f2fc004124efabd73cbef56710ac9 P 553c7127b2b446729881acd33d89da44: Deleting WAL directory at /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/minicluster-data/ts-2/wal/wals/fe3f2fc004124efabd73cbef56710ac9
I20260926 03:58:28.174772 2678 ts_tablet_manager.cc:1939] T fe3f2fc004124efabd73cbef56710ac9 P 553c7127b2b446729881acd33d89da44: Deleting consensus metadata
I20260926 03:58:28.175065 2639 tablet_bootstrap.cc:492] T c328a979844741a6b1e7daee9d1cfaca P 830c2976e7ef4eb4a96416333fcf1aa1: Bootstrap starting.
I20260926 03:58:28.175171 2116 catalog_manager.cc:4841] TS 553c7127b2b446729881acd33d89da44 (127.0.98.3:35577): tablet fe3f2fc004124efabd73cbef56710ac9 (table test-table [id=61eebd3e319541018adc5d24d7865279]) successfully deleted
I20260926 03:58:28.175328 2634 tablet_bootstrap.cc:492] T c328a979844741a6b1e7daee9d1cfaca P 553c7127b2b446729881acd33d89da44: No bootstrap required, opened a new log
I20260926 03:58:28.175366 2634 ts_tablet_manager.cc:1397] T c328a979844741a6b1e7daee9d1cfaca P 553c7127b2b446729881acd33d89da44: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 03:58:28.175487 2639 tablet_bootstrap.cc:654] T c328a979844741a6b1e7daee9d1cfaca P 830c2976e7ef4eb4a96416333fcf1aa1: Neither blocks nor log segments found. Creating new log.
I20260926 03:58:28.175506 2634 raft_consensus.cc:348] T c328a979844741a6b1e7daee9d1cfaca P 553c7127b2b446729881acd33d89da44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "216d653228874c8bb95fa8cfdda70c04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33193 } } peers { permanent_uuid: "830c2976e7ef4eb4a96416333fcf1aa1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44451 } } peers { permanent_uuid: "553c7127b2b446729881acd33d89da44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35577 } }
I20260926 03:58:28.175580 2634 raft_consensus.cc:374] T c328a979844741a6b1e7daee9d1cfaca P 553c7127b2b446729881acd33d89da44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:58:28.175607 2634 raft_consensus.cc:729] T c328a979844741a6b1e7daee9d1cfaca P 553c7127b2b446729881acd33d89da44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 553c7127b2b446729881acd33d89da44, State: Initialized, Role: FOLLOWER
I20260926 03:58:28.175663 2634 consensus_queue.cc:260] T c328a979844741a6b1e7daee9d1cfaca P 553c7127b2b446729881acd33d89da44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "216d653228874c8bb95fa8cfdda70c04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33193 } } peers { permanent_uuid: "830c2976e7ef4eb4a96416333fcf1aa1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44451 } } peers { permanent_uuid: "553c7127b2b446729881acd33d89da44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35577 } }
I20260926 03:58:28.175781 2634 ts_tablet_manager.cc:1428] T c328a979844741a6b1e7daee9d1cfaca P 553c7127b2b446729881acd33d89da44: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:28.176620 2639 tablet_bootstrap.cc:492] T c328a979844741a6b1e7daee9d1cfaca P 830c2976e7ef4eb4a96416333fcf1aa1: No bootstrap required, opened a new log
I20260926 03:58:28.176690 2639 ts_tablet_manager.cc:1397] T c328a979844741a6b1e7daee9d1cfaca P 830c2976e7ef4eb4a96416333fcf1aa1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 03:58:28.176709 2676 ts_tablet_manager.cc:1918] T fe3f2fc004124efabd73cbef56710ac9 P 216d653228874c8bb95fa8cfdda70c04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 03:58:28.176774 2676 log.cc:1196] T fe3f2fc004124efabd73cbef56710ac9 P 216d653228874c8bb95fa8cfdda70c04: Deleting WAL directory at /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/minicluster-data/ts-0/wal/wals/fe3f2fc004124efabd73cbef56710ac9
I20260926 03:58:28.176812 2639 raft_consensus.cc:348] T c328a979844741a6b1e7daee9d1cfaca P 830c2976e7ef4eb4a96416333fcf1aa1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "216d653228874c8bb95fa8cfdda70c04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33193 } } peers { permanent_uuid: "830c2976e7ef4eb4a96416333fcf1aa1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44451 } } peers { permanent_uuid: "553c7127b2b446729881acd33d89da44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35577 } }
I20260926 03:58:28.176869 2639 raft_consensus.cc:374] T c328a979844741a6b1e7daee9d1cfaca P 830c2976e7ef4eb4a96416333fcf1aa1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:58:28.176888 2639 raft_consensus.cc:729] T c328a979844741a6b1e7daee9d1cfaca P 830c2976e7ef4eb4a96416333fcf1aa1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 830c2976e7ef4eb4a96416333fcf1aa1, State: Initialized, Role: FOLLOWER
I20260926 03:58:28.176926 2639 consensus_queue.cc:260] T c328a979844741a6b1e7daee9d1cfaca P 830c2976e7ef4eb4a96416333fcf1aa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "216d653228874c8bb95fa8cfdda70c04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33193 } } peers { permanent_uuid: "830c2976e7ef4eb4a96416333fcf1aa1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44451 } } peers { permanent_uuid: "553c7127b2b446729881acd33d89da44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35577 } }
I20260926 03:58:28.177009 2639 ts_tablet_manager.cc:1428] T c328a979844741a6b1e7daee9d1cfaca P 830c2976e7ef4eb4a96416333fcf1aa1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:28.177021 2676 ts_tablet_manager.cc:1939] T fe3f2fc004124efabd73cbef56710ac9 P 216d653228874c8bb95fa8cfdda70c04: Deleting consensus metadata
I20260926 03:58:28.177161 2635 tablet_bootstrap.cc:492] T c328a979844741a6b1e7daee9d1cfaca P 216d653228874c8bb95fa8cfdda70c04: Bootstrap starting.
I20260926 03:58:28.177469 2635 tablet_bootstrap.cc:654] T c328a979844741a6b1e7daee9d1cfaca P 216d653228874c8bb95fa8cfdda70c04: Neither blocks nor log segments found. Creating new log.
I20260926 03:58:28.177878 2115 catalog_manager.cc:4841] TS 216d653228874c8bb95fa8cfdda70c04 (127.0.98.1:33193): tablet fe3f2fc004124efabd73cbef56710ac9 (table test-table [id=61eebd3e319541018adc5d24d7865279]) successfully deleted
I20260926 03:58:28.178028 2635 tablet_bootstrap.cc:492] T c328a979844741a6b1e7daee9d1cfaca P 216d653228874c8bb95fa8cfdda70c04: No bootstrap required, opened a new log
I20260926 03:58:28.178099 2635 ts_tablet_manager.cc:1397] T c328a979844741a6b1e7daee9d1cfaca P 216d653228874c8bb95fa8cfdda70c04: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 03:58:28.178246 2635 raft_consensus.cc:348] T c328a979844741a6b1e7daee9d1cfaca P 216d653228874c8bb95fa8cfdda70c04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "216d653228874c8bb95fa8cfdda70c04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33193 } } peers { permanent_uuid: "830c2976e7ef4eb4a96416333fcf1aa1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44451 } } peers { permanent_uuid: "553c7127b2b446729881acd33d89da44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35577 } }
I20260926 03:58:28.178320 2635 raft_consensus.cc:374] T c328a979844741a6b1e7daee9d1cfaca P 216d653228874c8bb95fa8cfdda70c04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:58:28.178345 2635 raft_consensus.cc:729] T c328a979844741a6b1e7daee9d1cfaca P 216d653228874c8bb95fa8cfdda70c04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 216d653228874c8bb95fa8cfdda70c04, State: Initialized, Role: FOLLOWER
I20260926 03:58:28.178388 2635 consensus_queue.cc:260] T c328a979844741a6b1e7daee9d1cfaca P 216d653228874c8bb95fa8cfdda70c04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "216d653228874c8bb95fa8cfdda70c04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33193 } } peers { permanent_uuid: "830c2976e7ef4eb4a96416333fcf1aa1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44451 } } peers { permanent_uuid: "553c7127b2b446729881acd33d89da44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35577 } }
I20260926 03:58:28.178488 2635 ts_tablet_manager.cc:1428] T c328a979844741a6b1e7daee9d1cfaca P 216d653228874c8bb95fa8cfdda70c04: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:28.248785 2638 raft_consensus.cc:482] T c328a979844741a6b1e7daee9d1cfaca P 553c7127b2b446729881acd33d89da44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:58:28.248896 2638 raft_consensus.cc:504] T c328a979844741a6b1e7daee9d1cfaca P 553c7127b2b446729881acd33d89da44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "216d653228874c8bb95fa8cfdda70c04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33193 } } peers { permanent_uuid: "830c2976e7ef4eb4a96416333fcf1aa1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44451 } } peers { permanent_uuid: "553c7127b2b446729881acd33d89da44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35577 } }
I20260926 03:58:28.249079 2638 leader_election.cc:288] T c328a979844741a6b1e7daee9d1cfaca P 553c7127b2b446729881acd33d89da44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 216d653228874c8bb95fa8cfdda70c04 (127.0.98.1:33193), 830c2976e7ef4eb4a96416333fcf1aa1 (127.0.98.2:44451)
I20260926 03:58:28.249246 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c328a979844741a6b1e7daee9d1cfaca" candidate_uuid: "553c7127b2b446729881acd33d89da44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "216d653228874c8bb95fa8cfdda70c04" is_pre_election: true
I20260926 03:58:28.249352 2288 raft_consensus.cc:2454] T c328a979844741a6b1e7daee9d1cfaca P 216d653228874c8bb95fa8cfdda70c04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 553c7127b2b446729881acd33d89da44 in term 0.
I20260926 03:58:28.249313 2428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c328a979844741a6b1e7daee9d1cfaca" candidate_uuid: "553c7127b2b446729881acd33d89da44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "830c2976e7ef4eb4a96416333fcf1aa1" is_pre_election: true
I20260926 03:58:28.249392 2428 raft_consensus.cc:2454] T c328a979844741a6b1e7daee9d1cfaca P 830c2976e7ef4eb4a96416333fcf1aa1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 553c7127b2b446729881acd33d89da44 in term 0.
I20260926 03:58:28.249498 2499 leader_election.cc:302] T c328a979844741a6b1e7daee9d1cfaca P 553c7127b2b446729881acd33d89da44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 216d653228874c8bb95fa8cfdda70c04, 553c7127b2b446729881acd33d89da44; no voters:
I20260926 03:58:28.249578 2638 raft_consensus.cc:2781] T c328a979844741a6b1e7daee9d1cfaca P 553c7127b2b446729881acd33d89da44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:58:28.249608 2638 raft_consensus.cc:482] T c328a979844741a6b1e7daee9d1cfaca P 553c7127b2b446729881acd33d89da44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:58:28.249642 2638 raft_consensus.cc:3037] T c328a979844741a6b1e7daee9d1cfaca P 553c7127b2b446729881acd33d89da44 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:58:28.250344 2638 raft_consensus.cc:504] T c328a979844741a6b1e7daee9d1cfaca P 553c7127b2b446729881acd33d89da44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "216d653228874c8bb95fa8cfdda70c04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33193 } } peers { permanent_uuid: "830c2976e7ef4eb4a96416333fcf1aa1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44451 } } peers { permanent_uuid: "553c7127b2b446729881acd33d89da44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35577 } }
I20260926 03:58:28.250486 2638 leader_election.cc:288] T c328a979844741a6b1e7daee9d1cfaca P 553c7127b2b446729881acd33d89da44 [CANDIDATE]: Term 1 election: Requested vote from peers 216d653228874c8bb95fa8cfdda70c04 (127.0.98.1:33193), 830c2976e7ef4eb4a96416333fcf1aa1 (127.0.98.2:44451)
I20260926 03:58:28.250625 2428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c328a979844741a6b1e7daee9d1cfaca" candidate_uuid: "553c7127b2b446729881acd33d89da44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "830c2976e7ef4eb4a96416333fcf1aa1"
I20260926 03:58:28.250641 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c328a979844741a6b1e7daee9d1cfaca" candidate_uuid: "553c7127b2b446729881acd33d89da44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "216d653228874c8bb95fa8cfdda70c04"
I20260926 03:58:28.250725 2428 raft_consensus.cc:3037] T c328a979844741a6b1e7daee9d1cfaca P 830c2976e7ef4eb4a96416333fcf1aa1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:58:28.250728 2288 raft_consensus.cc:3037] T c328a979844741a6b1e7daee9d1cfaca P 216d653228874c8bb95fa8cfdda70c04 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:58:28.251302 2428 raft_consensus.cc:2454] T c328a979844741a6b1e7daee9d1cfaca P 830c2976e7ef4eb4a96416333fcf1aa1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 553c7127b2b446729881acd33d89da44 in term 1.
I20260926 03:58:28.251475 2500 leader_election.cc:302] T c328a979844741a6b1e7daee9d1cfaca P 553c7127b2b446729881acd33d89da44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 553c7127b2b446729881acd33d89da44, 830c2976e7ef4eb4a96416333fcf1aa1; no voters:
I20260926 03:58:28.251528 2288 raft_consensus.cc:2454] T c328a979844741a6b1e7daee9d1cfaca P 216d653228874c8bb95fa8cfdda70c04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 553c7127b2b446729881acd33d89da44 in term 1.
I20260926 03:58:28.251576 2638 raft_consensus.cc:2781] T c328a979844741a6b1e7daee9d1cfaca P 553c7127b2b446729881acd33d89da44 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:58:28.251675 2638 raft_consensus.cc:686] T c328a979844741a6b1e7daee9d1cfaca P 553c7127b2b446729881acd33d89da44 [term 1 LEADER]: Becoming Leader. State: Replica: 553c7127b2b446729881acd33d89da44, State: Running, Role: LEADER
I20260926 03:58:28.251749 2638 consensus_queue.cc:237] T c328a979844741a6b1e7daee9d1cfaca P 553c7127b2b446729881acd33d89da44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "216d653228874c8bb95fa8cfdda70c04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33193 } } peers { permanent_uuid: "830c2976e7ef4eb4a96416333fcf1aa1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44451 } } peers { permanent_uuid: "553c7127b2b446729881acd33d89da44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35577 } }
I20260926 03:58:28.252507 2130 catalog_manager.cc:5495] T c328a979844741a6b1e7daee9d1cfaca P 553c7127b2b446729881acd33d89da44 reported cstate change: term changed from 0 to 1, leader changed from <none> to 553c7127b2b446729881acd33d89da44 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "553c7127b2b446729881acd33d89da44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "216d653228874c8bb95fa8cfdda70c04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "830c2976e7ef4eb4a96416333fcf1aa1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44451 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "553c7127b2b446729881acd33d89da44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35577 } health_report { overall_health: HEALTHY } } }
I20260926 03:58:28.270030 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:28.267453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59400 (local address 127.0.98.3:35577)
0926 03:58:28.267496 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:28.267498 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:28.267507 (+ 9us) server_negotiation.cc:407] Connection header received
0926 03:58:28.267576 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:28.267577 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:28.267596 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:28.267696 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:58:28.267796 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:28.268275 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:28.269171 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:28.269379 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:28.269532 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:28.269667 (+ 135us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:58:28.269784 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:58:28.269869 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:28.269908 (+ 39us) server_negotiation.cc:299] Negotiation successful
0926 03:58:28.269932 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 03:58:28.275156 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:28.272994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58589 (local address 127.0.98.1:33193)
0926 03:58:28.273032 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:28.273034 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:28.273043 (+ 9us) server_negotiation.cc:407] Connection header received
0926 03:58:28.273103 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:28.273103 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:28.273113 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:28.273167 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:58:28.273293 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:28.273737 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:28.274476 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:28.274682 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:28.274756 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:28.274917 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:58:28.274950 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:58:28.275021 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:28.275073 (+ 52us) server_negotiation.cc:299] Negotiation successful
0926 03:58:28.275091 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 03:58:28.275174 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:28.272906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33193 (local address 127.0.98.3:58589)
0926 03:58:28.272990 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:28.273002 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:28.273048 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:28.273169 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:28.273170 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:28.273171 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 03:58:28.273229 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:28.273230 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:28.273815 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:28.273816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:28.274387 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:28.274388 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:28.274755 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:28.274756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:28.274818 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:28.274831 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 03:58:28.275020 (+ 189us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 03:58:28.275022 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 03:58:28.275071 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 03:58:28.275111 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260926 03:58:28.277129 2288 raft_consensus.cc:1261] T c328a979844741a6b1e7daee9d1cfaca P 216d653228874c8bb95fa8cfdda70c04 [term 1 FOLLOWER]: Refusing update from remote peer 553c7127b2b446729881acd33d89da44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:58:28.277284 2428 raft_consensus.cc:1261] T c328a979844741a6b1e7daee9d1cfaca P 830c2976e7ef4eb4a96416333fcf1aa1 [term 1 FOLLOWER]: Refusing update from remote peer 553c7127b2b446729881acd33d89da44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:58:28.277395 2682 consensus_queue.cc:1035] T c328a979844741a6b1e7daee9d1cfaca P 553c7127b2b446729881acd33d89da44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "216d653228874c8bb95fa8cfdda70c04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:58:28.277505 2638 consensus_queue.cc:1035] T c328a979844741a6b1e7daee9d1cfaca P 553c7127b2b446729881acd33d89da44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "830c2976e7ef4eb4a96416333fcf1aa1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44451 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:58:28.285277 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:28.282593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37301 (local address 127.0.98.3:35577)
0926 03:58:28.282632 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:28.282633 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:28.282642 (+ 9us) server_negotiation.cc:407] Connection header received
0926 03:58:28.282684 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:28.282685 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:28.282697 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:28.282751 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:58:28.282880 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:28.283515 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:28.284536 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:28.284892 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:28.284961 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:28.285098 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:58:28.285123 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:58:28.285158 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:28.285214 (+ 56us) server_negotiation.cc:299] Negotiation successful
0926 03:58:28.285228 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 03:58:28.285359 2671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:28.282496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35577 (local address 127.0.98.1:37301)
0926 03:58:28.282581 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:28.282592 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:28.282629 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:28.282774 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:28.282775 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:28.282776 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 03:58:28.282818 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:28.282819 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:28.283595 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:28.283596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:28.284464 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:28.284467 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:28.284971 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:28.284972 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:28.285030 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:28.285037 (+ 7us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 03:58:28.285163 (+ 126us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 03:58:28.285165 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 03:58:28.285212 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 03:58:28.285252 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
W20260926 03:58:28.285683 2661 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 03:58:29.811033 2130 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43364:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:58:29.811160 2130 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43364:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:58:29.811959 2130 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d3414598d0e6464bb4b6f634282fb92a: Sending DeleteTablet for 3 replicas of tablet c328a979844741a6b1e7daee9d1cfaca
I20260926 03:58:29.812265 2408 tablet_service.cc:1514] Processing DeleteTablet for tablet c328a979844741a6b1e7daee9d1cfaca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:58:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55982
I20260926 03:58:29.812285 2268 tablet_service.cc:1514] Processing DeleteTablet for tablet c328a979844741a6b1e7daee9d1cfaca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:58:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42872
I20260926 03:58:29.812469 2548 tablet_service.cc:1514] Processing DeleteTablet for tablet c328a979844741a6b1e7daee9d1cfaca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:58:29 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59398
I20260926 03:58:29.812690 2714 tablet_replica.cc:330] T c328a979844741a6b1e7daee9d1cfaca P 553c7127b2b446729881acd33d89da44: stopping tablet replica
I20260926 03:58:29.812748 2714 raft_consensus.cc:2229] T c328a979844741a6b1e7daee9d1cfaca P 553c7127b2b446729881acd33d89da44 [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:58:29.812814 2714 raft_consensus.cc:2258] T c328a979844741a6b1e7daee9d1cfaca P 553c7127b2b446729881acd33d89da44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:58:29.812820 2712 tablet_replica.cc:330] T c328a979844741a6b1e7daee9d1cfaca P 216d653228874c8bb95fa8cfdda70c04: stopping tablet replica
I20260926 03:58:29.812881 2712 raft_consensus.cc:2229] T c328a979844741a6b1e7daee9d1cfaca P 216d653228874c8bb95fa8cfdda70c04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:58:29.812918 2712 raft_consensus.cc:2258] T c328a979844741a6b1e7daee9d1cfaca P 216d653228874c8bb95fa8cfdda70c04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:58:29.813040 2714 ts_tablet_manager.cc:1905] T c328a979844741a6b1e7daee9d1cfaca P 553c7127b2b446729881acd33d89da44: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:58:29.813079 2713 tablet_replica.cc:330] T c328a979844741a6b1e7daee9d1cfaca P 830c2976e7ef4eb4a96416333fcf1aa1: stopping tablet replica
I20260926 03:58:29.813136 2713 raft_consensus.cc:2229] T c328a979844741a6b1e7daee9d1cfaca P 830c2976e7ef4eb4a96416333fcf1aa1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:58:29.813172 2713 raft_consensus.cc:2258] T c328a979844741a6b1e7daee9d1cfaca P 830c2976e7ef4eb4a96416333fcf1aa1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:58:29.813395 2713 ts_tablet_manager.cc:1905] T c328a979844741a6b1e7daee9d1cfaca P 830c2976e7ef4eb4a96416333fcf1aa1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:58:29.814869 2714 ts_tablet_manager.cc:1918] T c328a979844741a6b1e7daee9d1cfaca P 553c7127b2b446729881acd33d89da44: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 03:58:29.814934 2714 log.cc:1196] T c328a979844741a6b1e7daee9d1cfaca P 553c7127b2b446729881acd33d89da44: Deleting WAL directory at /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/minicluster-data/ts-2/wal/wals/c328a979844741a6b1e7daee9d1cfaca
I20260926 03:58:29.815169 2713 ts_tablet_manager.cc:1918] T c328a979844741a6b1e7daee9d1cfaca P 830c2976e7ef4eb4a96416333fcf1aa1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 03:58:29.815188 2714 ts_tablet_manager.cc:1939] T c328a979844741a6b1e7daee9d1cfaca P 553c7127b2b446729881acd33d89da44: Deleting consensus metadata
I20260926 03:58:29.815248 2713 log.cc:1196] T c328a979844741a6b1e7daee9d1cfaca P 830c2976e7ef4eb4a96416333fcf1aa1: Deleting WAL directory at /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/minicluster-data/ts-1/wal/wals/c328a979844741a6b1e7daee9d1cfaca
I20260926 03:58:29.815449 2713 ts_tablet_manager.cc:1939] T c328a979844741a6b1e7daee9d1cfaca P 830c2976e7ef4eb4a96416333fcf1aa1: Deleting consensus metadata
I20260926 03:58:29.815536 2116 catalog_manager.cc:4841] TS 553c7127b2b446729881acd33d89da44 (127.0.98.3:35577): tablet c328a979844741a6b1e7daee9d1cfaca (table test-table [id=e9ff458e976e490c9c6a4dcc0436e05f]) successfully deleted
I20260926 03:58:29.815701 2712 ts_tablet_manager.cc:1905] T c328a979844741a6b1e7daee9d1cfaca P 216d653228874c8bb95fa8cfdda70c04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:58:29.815938 2116 catalog_manager.cc:4841] TS 830c2976e7ef4eb4a96416333fcf1aa1 (127.0.98.2:44451): tablet c328a979844741a6b1e7daee9d1cfaca (table test-table [id=e9ff458e976e490c9c6a4dcc0436e05f]) successfully deleted
I20260926 03:58:29.817013 2712 ts_tablet_manager.cc:1918] T c328a979844741a6b1e7daee9d1cfaca P 216d653228874c8bb95fa8cfdda70c04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 03:58:29.817090 2712 log.cc:1196] T c328a979844741a6b1e7daee9d1cfaca P 216d653228874c8bb95fa8cfdda70c04: Deleting WAL directory at /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/minicluster-data/ts-0/wal/wals/c328a979844741a6b1e7daee9d1cfaca
I20260926 03:58:29.817286 2712 ts_tablet_manager.cc:1939] T c328a979844741a6b1e7daee9d1cfaca P 216d653228874c8bb95fa8cfdda70c04: Deleting consensus metadata
I20260926 03:58:29.817646 2115 catalog_manager.cc:4841] TS 216d653228874c8bb95fa8cfdda70c04 (127.0.98.1:33193): tablet c328a979844741a6b1e7daee9d1cfaca (table test-table [id=e9ff458e976e490c9c6a4dcc0436e05f]) successfully deleted
Sep 26 03:58:29 dist-test-slave-nhpw krb5kdc[2086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395109, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 03:58:29 dist-test-slave-nhpw krb5kdc[2086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395109, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 03:58:29.841871 2722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:29.830774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43382 (local address 127.0.98.62:42475)
0926 03:58:29.830929 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:29.830930 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:29.830941 (+ 11us) server_negotiation.cc:407] Connection header received
0926 03:58:29.830993 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:29.830994 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:29.831009 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:29.831078 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:29.831611 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:29.832138 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:29.832937 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:29.833139 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:29.833205 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:29.839575 (+ 6370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:29.839576 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:29.839577 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:29.839583 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:29.840952 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:29.841257 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:29.841258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:29.841258 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:29.841320 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:29.841477 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:29.841478 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:29.841478 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:29.841583 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:29.841657 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:29.841740 (+ 83us) server_negotiation.cc:299] Negotiation successful
0926 03:58:29.841799 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":100,"threads_started":1}
I20260926 03:58:29.843561 2130 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43382:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:58:29.843744 2130 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:58:29.845160 2268 tablet_service.cc:1467] Processing CreateTablet for tablet ee5fc6c6ef9c46f8990657a71bbb8e61 (DEFAULT_TABLE table=test-table [id=09508cfb2e3a4764b6e39956ce92fc55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:58:29.845194 2408 tablet_service.cc:1467] Processing CreateTablet for tablet ee5fc6c6ef9c46f8990657a71bbb8e61 (DEFAULT_TABLE table=test-table [id=09508cfb2e3a4764b6e39956ce92fc55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:58:29.845225 2548 tablet_service.cc:1467] Processing CreateTablet for tablet ee5fc6c6ef9c46f8990657a71bbb8e61 (DEFAULT_TABLE table=test-table [id=09508cfb2e3a4764b6e39956ce92fc55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:58:29.845297 2268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee5fc6c6ef9c46f8990657a71bbb8e61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:58:29.845296 2408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee5fc6c6ef9c46f8990657a71bbb8e61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:58:29.845356 2548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee5fc6c6ef9c46f8990657a71bbb8e61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:58:29.846704 2724 tablet_bootstrap.cc:492] T ee5fc6c6ef9c46f8990657a71bbb8e61 P 830c2976e7ef4eb4a96416333fcf1aa1: Bootstrap starting.
I20260926 03:58:29.846757 2723 tablet_bootstrap.cc:492] T ee5fc6c6ef9c46f8990657a71bbb8e61 P 553c7127b2b446729881acd33d89da44: Bootstrap starting.
I20260926 03:58:29.846905 2725 tablet_bootstrap.cc:492] T ee5fc6c6ef9c46f8990657a71bbb8e61 P 216d653228874c8bb95fa8cfdda70c04: Bootstrap starting.
I20260926 03:58:29.847131 2724 tablet_bootstrap.cc:654] T ee5fc6c6ef9c46f8990657a71bbb8e61 P 830c2976e7ef4eb4a96416333fcf1aa1: Neither blocks nor log segments found. Creating new log.
I20260926 03:58:29.847152 2723 tablet_bootstrap.cc:654] T ee5fc6c6ef9c46f8990657a71bbb8e61 P 553c7127b2b446729881acd33d89da44: Neither blocks nor log segments found. Creating new log.
I20260926 03:58:29.847237 2725 tablet_bootstrap.cc:654] T ee5fc6c6ef9c46f8990657a71bbb8e61 P 216d653228874c8bb95fa8cfdda70c04: Neither blocks nor log segments found. Creating new log.
I20260926 03:58:29.847785 2725 tablet_bootstrap.cc:492] T ee5fc6c6ef9c46f8990657a71bbb8e61 P 216d653228874c8bb95fa8cfdda70c04: No bootstrap required, opened a new log
I20260926 03:58:29.847785 2723 tablet_bootstrap.cc:492] T ee5fc6c6ef9c46f8990657a71bbb8e61 P 553c7127b2b446729881acd33d89da44: No bootstrap required, opened a new log
I20260926 03:58:29.847838 2723 ts_tablet_manager.cc:1397] T ee5fc6c6ef9c46f8990657a71bbb8e61 P 553c7127b2b446729881acd33d89da44: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 03:58:29.847837 2725 ts_tablet_manager.cc:1397] T ee5fc6c6ef9c46f8990657a71bbb8e61 P 216d653228874c8bb95fa8cfdda70c04: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 03:58:29.847923 2724 tablet_bootstrap.cc:492] T ee5fc6c6ef9c46f8990657a71bbb8e61 P 830c2976e7ef4eb4a96416333fcf1aa1: No bootstrap required, opened a new log
I20260926 03:58:29.847975 2724 ts_tablet_manager.cc:1397] T ee5fc6c6ef9c46f8990657a71bbb8e61 P 830c2976e7ef4eb4a96416333fcf1aa1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 03:58:29.847957 2725 raft_consensus.cc:348] T ee5fc6c6ef9c46f8990657a71bbb8e61 P 216d653228874c8bb95fa8cfdda70c04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "216d653228874c8bb95fa8cfdda70c04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33193 } } peers { permanent_uuid: "830c2976e7ef4eb4a96416333fcf1aa1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44451 } } peers { permanent_uuid: "553c7127b2b446729881acd33d89da44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35577 } }
I20260926 03:58:29.847957 2723 raft_consensus.cc:348] T ee5fc6c6ef9c46f8990657a71bbb8e61 P 553c7127b2b446729881acd33d89da44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "216d653228874c8bb95fa8cfdda70c04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33193 } } peers { permanent_uuid: "830c2976e7ef4eb4a96416333fcf1aa1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44451 } } peers { permanent_uuid: "553c7127b2b446729881acd33d89da44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35577 } }
I20260926 03:58:29.848021 2725 raft_consensus.cc:374] T ee5fc6c6ef9c46f8990657a71bbb8e61 P 216d653228874c8bb95fa8cfdda70c04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:58:29.848021 2723 raft_consensus.cc:374] T ee5fc6c6ef9c46f8990657a71bbb8e61 P 553c7127b2b446729881acd33d89da44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:58:29.848049 2725 raft_consensus.cc:729] T ee5fc6c6ef9c46f8990657a71bbb8e61 P 216d653228874c8bb95fa8cfdda70c04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 216d653228874c8bb95fa8cfdda70c04, State: Initialized, Role: FOLLOWER
I20260926 03:58:29.848049 2723 raft_consensus.cc:729] T ee5fc6c6ef9c46f8990657a71bbb8e61 P 553c7127b2b446729881acd33d89da44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 553c7127b2b446729881acd33d89da44, State: Initialized, Role: FOLLOWER
I20260926 03:58:29.848101 2723 consensus_queue.cc:260] T ee5fc6c6ef9c46f8990657a71bbb8e61 P 553c7127b2b446729881acd33d89da44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "216d653228874c8bb95fa8cfdda70c04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33193 } } peers { permanent_uuid: "830c2976e7ef4eb4a96416333fcf1aa1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44451 } } peers { permanent_uuid: "553c7127b2b446729881acd33d89da44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35577 } }
I20260926 03:58:29.848132 2724 raft_consensus.cc:348] T ee5fc6c6ef9c46f8990657a71bbb8e61 P 830c2976e7ef4eb4a96416333fcf1aa1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "216d653228874c8bb95fa8cfdda70c04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33193 } } peers { permanent_uuid: "830c2976e7ef4eb4a96416333fcf1aa1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44451 } } peers { permanent_uuid: "553c7127b2b446729881acd33d89da44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35577 } }
I20260926 03:58:29.848201 2723 ts_tablet_manager.cc:1428] T ee5fc6c6ef9c46f8990657a71bbb8e61 P 553c7127b2b446729881acd33d89da44: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:29.848209 2724 raft_consensus.cc:374] T ee5fc6c6ef9c46f8990657a71bbb8e61 P 830c2976e7ef4eb4a96416333fcf1aa1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:58:29.848232 2724 raft_consensus.cc:729] T ee5fc6c6ef9c46f8990657a71bbb8e61 P 830c2976e7ef4eb4a96416333fcf1aa1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 830c2976e7ef4eb4a96416333fcf1aa1, State: Initialized, Role: FOLLOWER
I20260926 03:58:29.848275 2724 consensus_queue.cc:260] T ee5fc6c6ef9c46f8990657a71bbb8e61 P 830c2976e7ef4eb4a96416333fcf1aa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "216d653228874c8bb95fa8cfdda70c04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33193 } } peers { permanent_uuid: "830c2976e7ef4eb4a96416333fcf1aa1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44451 } } peers { permanent_uuid: "553c7127b2b446729881acd33d89da44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35577 } }
I20260926 03:58:29.848114 2725 consensus_queue.cc:260] T ee5fc6c6ef9c46f8990657a71bbb8e61 P 216d653228874c8bb95fa8cfdda70c04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "216d653228874c8bb95fa8cfdda70c04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33193 } } peers { permanent_uuid: "830c2976e7ef4eb4a96416333fcf1aa1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44451 } } peers { permanent_uuid: "553c7127b2b446729881acd33d89da44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35577 } }
I20260926 03:58:29.848436 2724 ts_tablet_manager.cc:1428] T ee5fc6c6ef9c46f8990657a71bbb8e61 P 830c2976e7ef4eb4a96416333fcf1aa1: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 03:58:29.848656 2725 ts_tablet_manager.cc:1428] T ee5fc6c6ef9c46f8990657a71bbb8e61 P 216d653228874c8bb95fa8cfdda70c04: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 03:58:29.975767 2729 raft_consensus.cc:482] T ee5fc6c6ef9c46f8990657a71bbb8e61 P 830c2976e7ef4eb4a96416333fcf1aa1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:58:29.975904 2729 raft_consensus.cc:504] T ee5fc6c6ef9c46f8990657a71bbb8e61 P 830c2976e7ef4eb4a96416333fcf1aa1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "216d653228874c8bb95fa8cfdda70c04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33193 } } peers { permanent_uuid: "830c2976e7ef4eb4a96416333fcf1aa1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44451 } } peers { permanent_uuid: "553c7127b2b446729881acd33d89da44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35577 } }
I20260926 03:58:29.976280 2729 leader_election.cc:288] T ee5fc6c6ef9c46f8990657a71bbb8e61 P 830c2976e7ef4eb4a96416333fcf1aa1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 216d653228874c8bb95fa8cfdda70c04 (127.0.98.1:33193), 553c7127b2b446729881acd33d89da44 (127.0.98.3:35577)
I20260926 03:58:29.980413 2733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:29.976708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43521 (local address 127.0.98.3:35577)
0926 03:58:29.976833 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:29.976834 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:29.976841 (+ 7us) server_negotiation.cc:407] Connection header received
0926 03:58:29.976867 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:29.976868 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:29.976878 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:29.976949 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:58:29.977114 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:29.977876 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:29.979459 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:29.980033 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:29.980116 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:29.980131 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:29.980275 (+ 144us) server_negotiation.cc:299] Negotiation successful
0926 03:58:29.980338 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":72,"threads_started":1}
I20260926 03:58:29.980413 2732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:29.976558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36621 (local address 127.0.98.1:33193)
0926 03:58:29.976728 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:29.976730 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:29.976769 (+ 39us) server_negotiation.cc:407] Connection header received
0926 03:58:29.976838 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:29.976839 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:29.976852 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:29.976945 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:58:29.977115 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:29.977908 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:29.979459 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:29.980038 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:29.980154 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:29.980165 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:29.980295 (+ 130us) server_negotiation.cc:299] Negotiation successful
0926 03:58:29.980338 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":94,"threads_started":1}
I20260926 03:58:29.980432 2731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:29.976487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33193 (local address 127.0.98.2:36621)
0926 03:58:29.976710 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:29.976719 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:29.976773 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:29.976966 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:29.976967 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:29.976971 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:58:29.977031 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:29.977033 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:29.977996 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:29.977997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:29.979346 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:29.979350 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:29.980152 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:29.980152 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:29.980200 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:29.980203 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 03:58:29.980255 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 03:58:29.980317 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":101,"thread_start_us":128,"threads_started":1}
I20260926 03:58:29.980439 2730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:29.976486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35577 (local address 127.0.98.2:43521)
0926 03:58:29.976692 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:29.976705 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:29.976773 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:29.976966 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:29.976967 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:29.976969 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:58:29.977032 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:29.977033 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:29.977996 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:29.977997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:29.979346 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:29.979350 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:29.980127 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:29.980128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:29.980191 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:29.980196 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 03:58:29.980255 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 03:58:29.980313 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":35,"thread_start_us":138,"threads_started":1}
I20260926 03:58:29.980737 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee5fc6c6ef9c46f8990657a71bbb8e61" candidate_uuid: "830c2976e7ef4eb4a96416333fcf1aa1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "216d653228874c8bb95fa8cfdda70c04" is_pre_election: true
I20260926 03:58:29.980840 2288 raft_consensus.cc:2454] T ee5fc6c6ef9c46f8990657a71bbb8e61 P 216d653228874c8bb95fa8cfdda70c04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 830c2976e7ef4eb4a96416333fcf1aa1 in term 0.
I20260926 03:58:29.981025 2359 leader_election.cc:302] T ee5fc6c6ef9c46f8990657a71bbb8e61 P 830c2976e7ef4eb4a96416333fcf1aa1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 216d653228874c8bb95fa8cfdda70c04, 830c2976e7ef4eb4a96416333fcf1aa1; no voters:
I20260926 03:58:29.981015 2568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee5fc6c6ef9c46f8990657a71bbb8e61" candidate_uuid: "830c2976e7ef4eb4a96416333fcf1aa1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "553c7127b2b446729881acd33d89da44" is_pre_election: true
I20260926 03:58:29.981120 2568 raft_consensus.cc:2454] T ee5fc6c6ef9c46f8990657a71bbb8e61 P 553c7127b2b446729881acd33d89da44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 830c2976e7ef4eb4a96416333fcf1aa1 in term 0.
I20260926 03:58:29.981161 2729 raft_consensus.cc:2781] T ee5fc6c6ef9c46f8990657a71bbb8e61 P 830c2976e7ef4eb4a96416333fcf1aa1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:58:29.981215 2729 raft_consensus.cc:482] T ee5fc6c6ef9c46f8990657a71bbb8e61 P 830c2976e7ef4eb4a96416333fcf1aa1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:58:29.981233 2729 raft_consensus.cc:3037] T ee5fc6c6ef9c46f8990657a71bbb8e61 P 830c2976e7ef4eb4a96416333fcf1aa1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:58:29.982110 2729 raft_consensus.cc:504] T ee5fc6c6ef9c46f8990657a71bbb8e61 P 830c2976e7ef4eb4a96416333fcf1aa1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "216d653228874c8bb95fa8cfdda70c04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33193 } } peers { permanent_uuid: "830c2976e7ef4eb4a96416333fcf1aa1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44451 } } peers { permanent_uuid: "553c7127b2b446729881acd33d89da44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35577 } }
I20260926 03:58:29.982281 2729 leader_election.cc:288] T ee5fc6c6ef9c46f8990657a71bbb8e61 P 830c2976e7ef4eb4a96416333fcf1aa1 [CANDIDATE]: Term 1 election: Requested vote from peers 216d653228874c8bb95fa8cfdda70c04 (127.0.98.1:33193), 553c7127b2b446729881acd33d89da44 (127.0.98.3:35577)
I20260926 03:58:29.982488 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee5fc6c6ef9c46f8990657a71bbb8e61" candidate_uuid: "830c2976e7ef4eb4a96416333fcf1aa1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "216d653228874c8bb95fa8cfdda70c04"
I20260926 03:58:29.982481 2568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee5fc6c6ef9c46f8990657a71bbb8e61" candidate_uuid: "830c2976e7ef4eb4a96416333fcf1aa1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "553c7127b2b446729881acd33d89da44"
I20260926 03:58:29.982559 2288 raft_consensus.cc:3037] T ee5fc6c6ef9c46f8990657a71bbb8e61 P 216d653228874c8bb95fa8cfdda70c04 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:58:29.982559 2568 raft_consensus.cc:3037] T ee5fc6c6ef9c46f8990657a71bbb8e61 P 553c7127b2b446729881acd33d89da44 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:58:29.983301 2568 raft_consensus.cc:2454] T ee5fc6c6ef9c46f8990657a71bbb8e61 P 553c7127b2b446729881acd33d89da44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 830c2976e7ef4eb4a96416333fcf1aa1 in term 1.
I20260926 03:58:29.983301 2288 raft_consensus.cc:2454] T ee5fc6c6ef9c46f8990657a71bbb8e61 P 216d653228874c8bb95fa8cfdda70c04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 830c2976e7ef4eb4a96416333fcf1aa1 in term 1.
I20260926 03:58:29.983548 2359 leader_election.cc:302] T ee5fc6c6ef9c46f8990657a71bbb8e61 P 830c2976e7ef4eb4a96416333fcf1aa1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 216d653228874c8bb95fa8cfdda70c04, 830c2976e7ef4eb4a96416333fcf1aa1; no voters:
I20260926 03:58:29.983641 2729 raft_consensus.cc:2781] T ee5fc6c6ef9c46f8990657a71bbb8e61 P 830c2976e7ef4eb4a96416333fcf1aa1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:58:29.983817 2729 raft_consensus.cc:686] T ee5fc6c6ef9c46f8990657a71bbb8e61 P 830c2976e7ef4eb4a96416333fcf1aa1 [term 1 LEADER]: Becoming Leader. State: Replica: 830c2976e7ef4eb4a96416333fcf1aa1, State: Running, Role: LEADER
I20260926 03:58:29.983911 2729 consensus_queue.cc:237] T ee5fc6c6ef9c46f8990657a71bbb8e61 P 830c2976e7ef4eb4a96416333fcf1aa1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "216d653228874c8bb95fa8cfdda70c04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33193 } } peers { permanent_uuid: "830c2976e7ef4eb4a96416333fcf1aa1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44451 } } peers { permanent_uuid: "553c7127b2b446729881acd33d89da44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35577 } }
I20260926 03:58:29.984750 2131 catalog_manager.cc:5495] T ee5fc6c6ef9c46f8990657a71bbb8e61 P 830c2976e7ef4eb4a96416333fcf1aa1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 830c2976e7ef4eb4a96416333fcf1aa1 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "830c2976e7ef4eb4a96416333fcf1aa1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "216d653228874c8bb95fa8cfdda70c04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "830c2976e7ef4eb4a96416333fcf1aa1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44451 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "553c7127b2b446729881acd33d89da44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35577 } health_report { overall_health: UNKNOWN } } }
I20260926 03:58:29.996042 2737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:29.993592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55990 (local address 127.0.98.2:44451)
0926 03:58:29.993739 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:29.993741 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:29.993749 (+ 8us) server_negotiation.cc:407] Connection header received
0926 03:58:29.993780 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:29.993780 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:29.993799 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:29.993879 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:58:29.994035 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:29.994540 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:29.995286 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:29.995513 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:29.995576 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:29.995709 (+ 133us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:58:29.995826 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:58:29.995898 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:29.995938 (+ 40us) server_negotiation.cc:299] Negotiation successful
0926 03:58:29.995956 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":76,"threads_started":1}
I20260926 03:58:30.000934 2732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:29.998556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44393 (local address 127.0.98.1:33193)
0926 03:58:29.998621 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:29.998623 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:29.998633 (+ 10us) server_negotiation.cc:407] Connection header received
0926 03:58:29.998702 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:29.998703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:29.998714 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:29.998767 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:58:29.998938 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:29.999393 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:30.000154 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:30.000382 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:30.000469 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:30.000668 (+ 199us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:58:30.000703 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:58:30.000774 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:30.000835 (+ 61us) server_negotiation.cc:299] Negotiation successful
0926 03:58:30.000856 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 03:58:30.000965 2740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:29.998402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33193 (local address 127.0.98.2:44393)
0926 03:58:29.998558 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:29.998567 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:29.998638 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:29.998784 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:29.998785 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:29.998787 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 03:58:29.998857 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:29.998859 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:29.999481 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:29.999482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:30.000069 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:30.000071 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:30.000473 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:30.000474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:30.000547 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:30.000559 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 03:58:30.000773 (+ 214us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 03:58:30.000775 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 03:58:30.000830 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 03:58:30.000883 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":89,"threads_started":1}
I20260926 03:58:30.003331 2568 raft_consensus.cc:1261] T ee5fc6c6ef9c46f8990657a71bbb8e61 P 553c7127b2b446729881acd33d89da44 [term 1 FOLLOWER]: Refusing update from remote peer 830c2976e7ef4eb4a96416333fcf1aa1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:58:30.003389 2288 raft_consensus.cc:1261] T ee5fc6c6ef9c46f8990657a71bbb8e61 P 216d653228874c8bb95fa8cfdda70c04 [term 1 FOLLOWER]: Refusing update from remote peer 830c2976e7ef4eb4a96416333fcf1aa1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:58:30.003609 2734 consensus_queue.cc:1035] T ee5fc6c6ef9c46f8990657a71bbb8e61 P 830c2976e7ef4eb4a96416333fcf1aa1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "553c7127b2b446729881acd33d89da44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:58:30.003755 2729 consensus_queue.cc:1035] T ee5fc6c6ef9c46f8990657a71bbb8e61 P 830c2976e7ef4eb4a96416333fcf1aa1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "216d653228874c8bb95fa8cfdda70c04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:58:30.011569 2737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:30.009057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51283 (local address 127.0.98.2:44451)
0926 03:58:30.009112 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:30.009113 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:30.009246 (+ 133us) server_negotiation.cc:407] Connection header received
0926 03:58:30.009307 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:30.009308 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:30.009320 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:30.009370 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:58:30.009544 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:30.010031 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:30.010792 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:30.011065 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:30.011132 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:30.011267 (+ 135us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:58:30.011319 (+ 52us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:58:30.011365 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:30.011470 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 03:58:30.011491 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 03:58:30.011574 2745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:30.009038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44451 (local address 127.0.98.1:51283)
0926 03:58:30.009185 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:30.009196 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:30.009253 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:30.009418 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:30.009419 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:30.009421 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 03:58:30.009467 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:30.009469 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:30.010117 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:30.010119 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:30.010690 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:30.010692 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:30.011131 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:30.011132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:30.011193 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:30.011199 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 03:58:30.011397 (+ 198us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 03:58:30.011399 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 03:58:30.011438 (+ 39us) client_negotiation.cc:240] Negotiation successful
0926 03:58:30.011491 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":99,"threads_started":1}
I20260926 03:58:30.026494 2131 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43382:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:58:30.026619 2131 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43382:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:58:30.027311 2131 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d3414598d0e6464bb4b6f634282fb92a: Sending DeleteTablet for 3 replicas of tablet ee5fc6c6ef9c46f8990657a71bbb8e61
I20260926 03:58:30.027566 2268 tablet_service.cc:1514] Processing DeleteTablet for tablet ee5fc6c6ef9c46f8990657a71bbb8e61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:58:30 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42872
I20260926 03:58:30.027680 2712 tablet_replica.cc:330] T ee5fc6c6ef9c46f8990657a71bbb8e61 P 216d653228874c8bb95fa8cfdda70c04: stopping tablet replica
I20260926 03:58:30.027740 2712 raft_consensus.cc:2229] T ee5fc6c6ef9c46f8990657a71bbb8e61 P 216d653228874c8bb95fa8cfdda70c04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:58:30.027786 2712 raft_consensus.cc:2258] T ee5fc6c6ef9c46f8990657a71bbb8e61 P 216d653228874c8bb95fa8cfdda70c04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:58:30.027822 2548 tablet_service.cc:1514] Processing DeleteTablet for tablet ee5fc6c6ef9c46f8990657a71bbb8e61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:58:30 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59398
I20260926 03:58:30.027937 2714 tablet_replica.cc:330] T ee5fc6c6ef9c46f8990657a71bbb8e61 P 553c7127b2b446729881acd33d89da44: stopping tablet replica
I20260926 03:58:30.027940 2408 tablet_service.cc:1514] Processing DeleteTablet for tablet ee5fc6c6ef9c46f8990657a71bbb8e61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:58:30 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55982
I20260926 03:58:30.027990 2714 raft_consensus.cc:2229] T ee5fc6c6ef9c46f8990657a71bbb8e61 P 553c7127b2b446729881acd33d89da44 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:58:30.028052 2714 raft_consensus.cc:2258] T ee5fc6c6ef9c46f8990657a71bbb8e61 P 553c7127b2b446729881acd33d89da44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:58:30.028064 2713 tablet_replica.cc:330] T ee5fc6c6ef9c46f8990657a71bbb8e61 P 830c2976e7ef4eb4a96416333fcf1aa1: stopping tablet replica
I20260926 03:58:30.028119 2713 raft_consensus.cc:2229] T ee5fc6c6ef9c46f8990657a71bbb8e61 P 830c2976e7ef4eb4a96416333fcf1aa1 [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:58:30.028214 2713 raft_consensus.cc:2258] T ee5fc6c6ef9c46f8990657a71bbb8e61 P 830c2976e7ef4eb4a96416333fcf1aa1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:58:30.028304 2712 ts_tablet_manager.cc:1905] T ee5fc6c6ef9c46f8990657a71bbb8e61 P 216d653228874c8bb95fa8cfdda70c04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:58:30.028384 2714 ts_tablet_manager.cc:1905] T ee5fc6c6ef9c46f8990657a71bbb8e61 P 553c7127b2b446729881acd33d89da44: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:58:30.028959 2713 ts_tablet_manager.cc:1905] T ee5fc6c6ef9c46f8990657a71bbb8e61 P 830c2976e7ef4eb4a96416333fcf1aa1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:58:30.028901 2131 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43382:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:58:30.029052 2131 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:58:30.030407 2712 ts_tablet_manager.cc:1918] T ee5fc6c6ef9c46f8990657a71bbb8e61 P 216d653228874c8bb95fa8cfdda70c04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 03:58:30.030416 2714 ts_tablet_manager.cc:1918] T ee5fc6c6ef9c46f8990657a71bbb8e61 P 553c7127b2b446729881acd33d89da44: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 03:58:30.030474 2712 log.cc:1196] T ee5fc6c6ef9c46f8990657a71bbb8e61 P 216d653228874c8bb95fa8cfdda70c04: Deleting WAL directory at /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/minicluster-data/ts-0/wal/wals/ee5fc6c6ef9c46f8990657a71bbb8e61
I20260926 03:58:30.030473 2714 log.cc:1196] T ee5fc6c6ef9c46f8990657a71bbb8e61 P 553c7127b2b446729881acd33d89da44: Deleting WAL directory at /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/minicluster-data/ts-2/wal/wals/ee5fc6c6ef9c46f8990657a71bbb8e61
I20260926 03:58:30.030570 2713 ts_tablet_manager.cc:1918] T ee5fc6c6ef9c46f8990657a71bbb8e61 P 830c2976e7ef4eb4a96416333fcf1aa1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 03:58:30.030624 2713 log.cc:1196] T ee5fc6c6ef9c46f8990657a71bbb8e61 P 830c2976e7ef4eb4a96416333fcf1aa1: Deleting WAL directory at /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/minicluster-data/ts-1/wal/wals/ee5fc6c6ef9c46f8990657a71bbb8e61
I20260926 03:58:30.030710 2712 ts_tablet_manager.cc:1939] T ee5fc6c6ef9c46f8990657a71bbb8e61 P 216d653228874c8bb95fa8cfdda70c04: Deleting consensus metadata
I20260926 03:58:30.030731 2714 ts_tablet_manager.cc:1939] T ee5fc6c6ef9c46f8990657a71bbb8e61 P 553c7127b2b446729881acd33d89da44: Deleting consensus metadata
I20260926 03:58:30.030838 2713 ts_tablet_manager.cc:1939] T ee5fc6c6ef9c46f8990657a71bbb8e61 P 830c2976e7ef4eb4a96416333fcf1aa1: Deleting consensus metadata
I20260926 03:58:30.031038 2115 catalog_manager.cc:4841] TS 216d653228874c8bb95fa8cfdda70c04 (127.0.98.1:33193): tablet ee5fc6c6ef9c46f8990657a71bbb8e61 (table test-table [id=09508cfb2e3a4764b6e39956ce92fc55]) successfully deleted
I20260926 03:58:30.031647 2268 tablet_service.cc:1467] Processing CreateTablet for tablet 8326d73ad09c479f8ca0eef383fca745 (DEFAULT_TABLE table=test-table [id=678f477103764114a60ddbe43391d1d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:58:30.031683 2116 catalog_manager.cc:4841] TS 830c2976e7ef4eb4a96416333fcf1aa1 (127.0.98.2:44451): tablet ee5fc6c6ef9c46f8990657a71bbb8e61 (table test-table [id=09508cfb2e3a4764b6e39956ce92fc55]) successfully deleted
I20260926 03:58:30.031737 2116 catalog_manager.cc:4841] TS 553c7127b2b446729881acd33d89da44 (127.0.98.3:35577): tablet ee5fc6c6ef9c46f8990657a71bbb8e61 (table test-table [id=09508cfb2e3a4764b6e39956ce92fc55]) successfully deleted
I20260926 03:58:30.031759 2268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8326d73ad09c479f8ca0eef383fca745. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:58:30.031905 2548 tablet_service.cc:1467] Processing CreateTablet for tablet 8326d73ad09c479f8ca0eef383fca745 (DEFAULT_TABLE table=test-table [id=678f477103764114a60ddbe43391d1d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:58:30.031911 2408 tablet_service.cc:1467] Processing CreateTablet for tablet 8326d73ad09c479f8ca0eef383fca745 (DEFAULT_TABLE table=test-table [id=678f477103764114a60ddbe43391d1d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:58:30.031988 2548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8326d73ad09c479f8ca0eef383fca745. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:58:30.032023 2408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8326d73ad09c479f8ca0eef383fca745. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:58:30.033109 2725 tablet_bootstrap.cc:492] T 8326d73ad09c479f8ca0eef383fca745 P 216d653228874c8bb95fa8cfdda70c04: Bootstrap starting.
I20260926 03:58:30.033149 2724 tablet_bootstrap.cc:492] T 8326d73ad09c479f8ca0eef383fca745 P 830c2976e7ef4eb4a96416333fcf1aa1: Bootstrap starting.
I20260926 03:58:30.033437 2725 tablet_bootstrap.cc:654] T 8326d73ad09c479f8ca0eef383fca745 P 216d653228874c8bb95fa8cfdda70c04: Neither blocks nor log segments found. Creating new log.
I20260926 03:58:30.033525 2724 tablet_bootstrap.cc:654] T 8326d73ad09c479f8ca0eef383fca745 P 830c2976e7ef4eb4a96416333fcf1aa1: Neither blocks nor log segments found. Creating new log.
I20260926 03:58:30.033764 2723 tablet_bootstrap.cc:492] T 8326d73ad09c479f8ca0eef383fca745 P 553c7127b2b446729881acd33d89da44: Bootstrap starting.
I20260926 03:58:30.034082 2723 tablet_bootstrap.cc:654] T 8326d73ad09c479f8ca0eef383fca745 P 553c7127b2b446729881acd33d89da44: Neither blocks nor log segments found. Creating new log.
I20260926 03:58:30.034459 2724 tablet_bootstrap.cc:492] T 8326d73ad09c479f8ca0eef383fca745 P 830c2976e7ef4eb4a96416333fcf1aa1: No bootstrap required, opened a new log
I20260926 03:58:30.034468 2725 tablet_bootstrap.cc:492] T 8326d73ad09c479f8ca0eef383fca745 P 216d653228874c8bb95fa8cfdda70c04: No bootstrap required, opened a new log
I20260926 03:58:30.034515 2724 ts_tablet_manager.cc:1397] T 8326d73ad09c479f8ca0eef383fca745 P 830c2976e7ef4eb4a96416333fcf1aa1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 03:58:30.034518 2725 ts_tablet_manager.cc:1397] T 8326d73ad09c479f8ca0eef383fca745 P 216d653228874c8bb95fa8cfdda70c04: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 03:58:30.034626 2725 raft_consensus.cc:348] T 8326d73ad09c479f8ca0eef383fca745 P 216d653228874c8bb95fa8cfdda70c04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "216d653228874c8bb95fa8cfdda70c04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33193 } } peers { permanent_uuid: "830c2976e7ef4eb4a96416333fcf1aa1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44451 } } peers { permanent_uuid: "553c7127b2b446729881acd33d89da44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35577 } }
I20260926 03:58:30.034665 2724 raft_consensus.cc:348] T 8326d73ad09c479f8ca0eef383fca745 P 830c2976e7ef4eb4a96416333fcf1aa1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "216d653228874c8bb95fa8cfdda70c04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33193 } } peers { permanent_uuid: "830c2976e7ef4eb4a96416333fcf1aa1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44451 } } peers { permanent_uuid: "553c7127b2b446729881acd33d89da44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35577 } }
I20260926 03:58:30.034713 2725 raft_consensus.cc:374] T 8326d73ad09c479f8ca0eef383fca745 P 216d653228874c8bb95fa8cfdda70c04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:58:30.034725 2724 raft_consensus.cc:374] T 8326d73ad09c479f8ca0eef383fca745 P 830c2976e7ef4eb4a96416333fcf1aa1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:58:30.034744 2724 raft_consensus.cc:729] T 8326d73ad09c479f8ca0eef383fca745 P 830c2976e7ef4eb4a96416333fcf1aa1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 830c2976e7ef4eb4a96416333fcf1aa1, State: Initialized, Role: FOLLOWER
I20260926 03:58:30.034744 2725 raft_consensus.cc:729] T 8326d73ad09c479f8ca0eef383fca745 P 216d653228874c8bb95fa8cfdda70c04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 216d653228874c8bb95fa8cfdda70c04, State: Initialized, Role: FOLLOWER
I20260926 03:58:30.034786 2725 consensus_queue.cc:260] T 8326d73ad09c479f8ca0eef383fca745 P 216d653228874c8bb95fa8cfdda70c04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "216d653228874c8bb95fa8cfdda70c04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33193 } } peers { permanent_uuid: "830c2976e7ef4eb4a96416333fcf1aa1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44451 } } peers { permanent_uuid: "553c7127b2b446729881acd33d89da44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35577 } }
I20260926 03:58:30.034796 2724 consensus_queue.cc:260] T 8326d73ad09c479f8ca0eef383fca745 P 830c2976e7ef4eb4a96416333fcf1aa1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "216d653228874c8bb95fa8cfdda70c04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33193 } } peers { permanent_uuid: "830c2976e7ef4eb4a96416333fcf1aa1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44451 } } peers { permanent_uuid: "553c7127b2b446729881acd33d89da44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35577 } }
I20260926 03:58:30.034888 2724 ts_tablet_manager.cc:1428] T 8326d73ad09c479f8ca0eef383fca745 P 830c2976e7ef4eb4a96416333fcf1aa1: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 03:58:30.034890 2725 ts_tablet_manager.cc:1428] T 8326d73ad09c479f8ca0eef383fca745 P 216d653228874c8bb95fa8cfdda70c04: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:30.035264 2723 tablet_bootstrap.cc:492] T 8326d73ad09c479f8ca0eef383fca745 P 553c7127b2b446729881acd33d89da44: No bootstrap required, opened a new log
I20260926 03:58:30.035332 2723 ts_tablet_manager.cc:1397] T 8326d73ad09c479f8ca0eef383fca745 P 553c7127b2b446729881acd33d89da44: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 03:58:30.035437 2723 raft_consensus.cc:348] T 8326d73ad09c479f8ca0eef383fca745 P 553c7127b2b446729881acd33d89da44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "216d653228874c8bb95fa8cfdda70c04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33193 } } peers { permanent_uuid: "830c2976e7ef4eb4a96416333fcf1aa1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44451 } } peers { permanent_uuid: "553c7127b2b446729881acd33d89da44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35577 } }
I20260926 03:58:30.035527 2723 raft_consensus.cc:374] T 8326d73ad09c479f8ca0eef383fca745 P 553c7127b2b446729881acd33d89da44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:58:30.035552 2723 raft_consensus.cc:729] T 8326d73ad09c479f8ca0eef383fca745 P 553c7127b2b446729881acd33d89da44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 553c7127b2b446729881acd33d89da44, State: Initialized, Role: FOLLOWER
I20260926 03:58:30.035594 2723 consensus_queue.cc:260] T 8326d73ad09c479f8ca0eef383fca745 P 553c7127b2b446729881acd33d89da44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "216d653228874c8bb95fa8cfdda70c04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33193 } } peers { permanent_uuid: "830c2976e7ef4eb4a96416333fcf1aa1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44451 } } peers { permanent_uuid: "553c7127b2b446729881acd33d89da44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35577 } }
I20260926 03:58:30.035691 2723 ts_tablet_manager.cc:1428] T 8326d73ad09c479f8ca0eef383fca745 P 553c7127b2b446729881acd33d89da44: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 03:58:30.253055 2743 raft_consensus.cc:482] T 8326d73ad09c479f8ca0eef383fca745 P 830c2976e7ef4eb4a96416333fcf1aa1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:58:30.253156 2743 raft_consensus.cc:504] T 8326d73ad09c479f8ca0eef383fca745 P 830c2976e7ef4eb4a96416333fcf1aa1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "216d653228874c8bb95fa8cfdda70c04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33193 } } peers { permanent_uuid: "830c2976e7ef4eb4a96416333fcf1aa1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44451 } } peers { permanent_uuid: "553c7127b2b446729881acd33d89da44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35577 } }
I20260926 03:58:30.253324 2743 leader_election.cc:288] T 8326d73ad09c479f8ca0eef383fca745 P 830c2976e7ef4eb4a96416333fcf1aa1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 216d653228874c8bb95fa8cfdda70c04 (127.0.98.1:33193), 553c7127b2b446729881acd33d89da44 (127.0.98.3:35577)
I20260926 03:58:30.253563 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8326d73ad09c479f8ca0eef383fca745" candidate_uuid: "830c2976e7ef4eb4a96416333fcf1aa1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "216d653228874c8bb95fa8cfdda70c04" is_pre_election: true
I20260926 03:58:30.253654 2288 raft_consensus.cc:2454] T 8326d73ad09c479f8ca0eef383fca745 P 216d653228874c8bb95fa8cfdda70c04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 830c2976e7ef4eb4a96416333fcf1aa1 in term 0.
I20260926 03:58:30.253572 2568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8326d73ad09c479f8ca0eef383fca745" candidate_uuid: "830c2976e7ef4eb4a96416333fcf1aa1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "553c7127b2b446729881acd33d89da44" is_pre_election: true
I20260926 03:58:30.253692 2568 raft_consensus.cc:2454] T 8326d73ad09c479f8ca0eef383fca745 P 553c7127b2b446729881acd33d89da44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 830c2976e7ef4eb4a96416333fcf1aa1 in term 0.
I20260926 03:58:30.253813 2359 leader_election.cc:302] T 8326d73ad09c479f8ca0eef383fca745 P 830c2976e7ef4eb4a96416333fcf1aa1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 216d653228874c8bb95fa8cfdda70c04, 830c2976e7ef4eb4a96416333fcf1aa1; no voters:
I20260926 03:58:30.253902 2743 raft_consensus.cc:2781] T 8326d73ad09c479f8ca0eef383fca745 P 830c2976e7ef4eb4a96416333fcf1aa1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:58:30.253937 2743 raft_consensus.cc:482] T 8326d73ad09c479f8ca0eef383fca745 P 830c2976e7ef4eb4a96416333fcf1aa1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:58:30.253960 2743 raft_consensus.cc:3037] T 8326d73ad09c479f8ca0eef383fca745 P 830c2976e7ef4eb4a96416333fcf1aa1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:58:30.254663 2743 raft_consensus.cc:504] T 8326d73ad09c479f8ca0eef383fca745 P 830c2976e7ef4eb4a96416333fcf1aa1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "216d653228874c8bb95fa8cfdda70c04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33193 } } peers { permanent_uuid: "830c2976e7ef4eb4a96416333fcf1aa1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44451 } } peers { permanent_uuid: "553c7127b2b446729881acd33d89da44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35577 } }
I20260926 03:58:30.254807 2743 leader_election.cc:288] T 8326d73ad09c479f8ca0eef383fca745 P 830c2976e7ef4eb4a96416333fcf1aa1 [CANDIDATE]: Term 1 election: Requested vote from peers 216d653228874c8bb95fa8cfdda70c04 (127.0.98.1:33193), 553c7127b2b446729881acd33d89da44 (127.0.98.3:35577)
I20260926 03:58:30.255017 2568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8326d73ad09c479f8ca0eef383fca745" candidate_uuid: "830c2976e7ef4eb4a96416333fcf1aa1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "553c7127b2b446729881acd33d89da44"
I20260926 03:58:30.255033 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8326d73ad09c479f8ca0eef383fca745" candidate_uuid: "830c2976e7ef4eb4a96416333fcf1aa1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "216d653228874c8bb95fa8cfdda70c04"
I20260926 03:58:30.255101 2568 raft_consensus.cc:3037] T 8326d73ad09c479f8ca0eef383fca745 P 553c7127b2b446729881acd33d89da44 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:58:30.255100 2288 raft_consensus.cc:3037] T 8326d73ad09c479f8ca0eef383fca745 P 216d653228874c8bb95fa8cfdda70c04 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:58:30.255862 2288 raft_consensus.cc:2454] T 8326d73ad09c479f8ca0eef383fca745 P 216d653228874c8bb95fa8cfdda70c04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 830c2976e7ef4eb4a96416333fcf1aa1 in term 1.
I20260926 03:58:30.255874 2568 raft_consensus.cc:2454] T 8326d73ad09c479f8ca0eef383fca745 P 553c7127b2b446729881acd33d89da44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 830c2976e7ef4eb4a96416333fcf1aa1 in term 1.
I20260926 03:58:30.256037 2360 leader_election.cc:302] T 8326d73ad09c479f8ca0eef383fca745 P 830c2976e7ef4eb4a96416333fcf1aa1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 553c7127b2b446729881acd33d89da44, 830c2976e7ef4eb4a96416333fcf1aa1; no voters:
I20260926 03:58:30.256160 2743 raft_consensus.cc:2781] T 8326d73ad09c479f8ca0eef383fca745 P 830c2976e7ef4eb4a96416333fcf1aa1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:58:30.256206 2743 raft_consensus.cc:686] T 8326d73ad09c479f8ca0eef383fca745 P 830c2976e7ef4eb4a96416333fcf1aa1 [term 1 LEADER]: Becoming Leader. State: Replica: 830c2976e7ef4eb4a96416333fcf1aa1, State: Running, Role: LEADER
I20260926 03:58:30.256253 2743 consensus_queue.cc:237] T 8326d73ad09c479f8ca0eef383fca745 P 830c2976e7ef4eb4a96416333fcf1aa1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "216d653228874c8bb95fa8cfdda70c04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33193 } } peers { permanent_uuid: "830c2976e7ef4eb4a96416333fcf1aa1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44451 } } peers { permanent_uuid: "553c7127b2b446729881acd33d89da44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35577 } }
I20260926 03:58:30.256811 2129 catalog_manager.cc:5495] T 8326d73ad09c479f8ca0eef383fca745 P 830c2976e7ef4eb4a96416333fcf1aa1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 830c2976e7ef4eb4a96416333fcf1aa1 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "830c2976e7ef4eb4a96416333fcf1aa1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "216d653228874c8bb95fa8cfdda70c04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33193 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "830c2976e7ef4eb4a96416333fcf1aa1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44451 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "553c7127b2b446729881acd33d89da44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35577 } health_report { overall_health: UNKNOWN } } }
I20260926 03:58:30.265993 2288 raft_consensus.cc:1261] T 8326d73ad09c479f8ca0eef383fca745 P 216d653228874c8bb95fa8cfdda70c04 [term 1 FOLLOWER]: Refusing update from remote peer 830c2976e7ef4eb4a96416333fcf1aa1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:58:30.265975 2568 raft_consensus.cc:1261] T 8326d73ad09c479f8ca0eef383fca745 P 553c7127b2b446729881acd33d89da44 [term 1 FOLLOWER]: Refusing update from remote peer 830c2976e7ef4eb4a96416333fcf1aa1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:58:30.266216 2743 consensus_queue.cc:1035] T 8326d73ad09c479f8ca0eef383fca745 P 830c2976e7ef4eb4a96416333fcf1aa1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "216d653228874c8bb95fa8cfdda70c04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:58:30.266330 2729 consensus_queue.cc:1035] T 8326d73ad09c479f8ca0eef383fca745 P 830c2976e7ef4eb4a96416333fcf1aa1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "553c7127b2b446729881acd33d89da44" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 03:58:30.270150 2696 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 03:58:31.790930 2131 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43382:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:58:31.791054 2131 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43382:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:58:31.791893 2131 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d3414598d0e6464bb4b6f634282fb92a: Sending DeleteTablet for 3 replicas of tablet 8326d73ad09c479f8ca0eef383fca745
I20260926 03:58:31.792171 2408 tablet_service.cc:1514] Processing DeleteTablet for tablet 8326d73ad09c479f8ca0eef383fca745 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:58:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55982
I20260926 03:58:31.792330 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 2203
I20260926 03:58:31.792354 2548 tablet_service.cc:1514] Processing DeleteTablet for tablet 8326d73ad09c479f8ca0eef383fca745 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:58:31 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59398
I20260926 03:58:31.793085 2774 tablet_replica.cc:330] T 8326d73ad09c479f8ca0eef383fca745 P 830c2976e7ef4eb4a96416333fcf1aa1: stopping tablet replica
I20260926 03:58:31.793195 2774 raft_consensus.cc:2229] T 8326d73ad09c479f8ca0eef383fca745 P 830c2976e7ef4eb4a96416333fcf1aa1 [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:58:31.793308 2774 raft_consensus.cc:2258] T 8326d73ad09c479f8ca0eef383fca745 P 830c2976e7ef4eb4a96416333fcf1aa1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:58:31.793481 2775 tablet_replica.cc:330] T 8326d73ad09c479f8ca0eef383fca745 P 553c7127b2b446729881acd33d89da44: stopping tablet replica
I20260926 03:58:31.793551 2775 raft_consensus.cc:2229] T 8326d73ad09c479f8ca0eef383fca745 P 553c7127b2b446729881acd33d89da44 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:58:31.793593 2775 raft_consensus.cc:2258] T 8326d73ad09c479f8ca0eef383fca745 P 553c7127b2b446729881acd33d89da44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:58:31.793769 2774 ts_tablet_manager.cc:1905] T 8326d73ad09c479f8ca0eef383fca745 P 830c2976e7ef4eb4a96416333fcf1aa1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:58:31.793826 2775 ts_tablet_manager.cc:1905] T 8326d73ad09c479f8ca0eef383fca745 P 553c7127b2b446729881acd33d89da44: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:58:31.795701 2775 ts_tablet_manager.cc:1918] T 8326d73ad09c479f8ca0eef383fca745 P 553c7127b2b446729881acd33d89da44: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 03:58:31.795701 2774 ts_tablet_manager.cc:1918] T 8326d73ad09c479f8ca0eef383fca745 P 830c2976e7ef4eb4a96416333fcf1aa1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 03:58:31.795768 2775 log.cc:1196] T 8326d73ad09c479f8ca0eef383fca745 P 553c7127b2b446729881acd33d89da44: Deleting WAL directory at /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/minicluster-data/ts-2/wal/wals/8326d73ad09c479f8ca0eef383fca745
I20260926 03:58:31.795768 2774 log.cc:1196] T 8326d73ad09c479f8ca0eef383fca745 P 830c2976e7ef4eb4a96416333fcf1aa1: Deleting WAL directory at /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790395095474643-392-0/minicluster-data/ts-1/wal/wals/8326d73ad09c479f8ca0eef383fca745
I20260926 03:58:31.796010 2774 ts_tablet_manager.cc:1939] T 8326d73ad09c479f8ca0eef383fca745 P 830c2976e7ef4eb4a96416333fcf1aa1: Deleting consensus metadata
I20260926 03:58:31.796031 2775 ts_tablet_manager.cc:1939] T 8326d73ad09c479f8ca0eef383fca745 P 553c7127b2b446729881acd33d89da44: Deleting consensus metadata
I20260926 03:58:31.796329 2116 catalog_manager.cc:4841] TS 830c2976e7ef4eb4a96416333fcf1aa1 (127.0.98.2:44451): tablet 8326d73ad09c479f8ca0eef383fca745 (table test-table [id=678f477103764114a60ddbe43391d1d9]) successfully deleted
I20260926 03:58:31.796413 2116 catalog_manager.cc:4841] TS 553c7127b2b446729881acd33d89da44 (127.0.98.3:35577): tablet 8326d73ad09c479f8ca0eef383fca745 (table test-table [id=678f477103764114a60ddbe43391d1d9]) successfully deleted
W20260926 03:58:31.798882 2115 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33193 (error 108)
I20260926 03:58:31.799103 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 2343
I20260926 03:58:31.799439 2777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:31.799255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33193 (local address 127.0.0.1:42894)
0926 03:58:31.799359 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:31.799399 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":53,"threads_started":1}
W20260926 03:58:31.799520 2115 catalog_manager.cc:4584] TS 216d653228874c8bb95fa8cfdda70c04 (127.0.98.1:33193): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8326d73ad09c479f8ca0eef383fca745: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33193: connect: Connection refused (error 111)
I20260926 03:58:31.805235 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 2483
I20260926 03:58:31.810792 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 2099
2026-09-26T03:58:31Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5685 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:58:31 dist-test-slave-nhpw krb5kdc[2780](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:58:31 dist-test-slave-nhpw krb5kdc[2780](info): set up 2 sockets
Sep 26 03:58:31 dist-test-slave-nhpw krb5kdc[2780](info): commencing operation
krb5kdc: starting...
W20260926 03:58:32.860456 392 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/krb5kdc/test-user.keytab.
Sep 26 03:58:32 dist-test-slave-nhpw krb5kdc[2780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:58:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:58:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 03:58:32.980237 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40013
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40013
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:58:33.058909 2793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:58:33.059173 2793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:58:33.059212 2793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:58:33.060410 2793 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:58:33.060497 2793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:58:33.060520 2793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:58:33.060542 2793 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:58:33.060570 2793 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:58:33.062129 2793 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40013
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40013
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2793
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:58:33.062414 2793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:58:33.062659 2793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:58:33.067646 2801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:33.067672 2799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:33.067710 2793 server_base.cc:1034] running on GCE node
W20260926 03:58:33.067724 2798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:33.068148 2793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:58:33.068486 2793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:58:33.069653 2793 hybrid_clock.cc:648] HybridClock initialized: now 1790395113069630 us; error 44 us; skew 500 ppm
Sep 26 03:58:33 dist-test-slave-nhpw krb5kdc[2780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395113, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:58:33.072208 2793 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 03:58:33.072772 2793 webserver.cc:460] Webserver started at http://127.0.98.62:43551/ using document root <none> and password file <none>
I20260926 03:58:33.073015 2793 fs_manager.cc:359] Metadata directory not provided
I20260926 03:58:33.073084 2793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:58:33.073226 2793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:58:33.074321 2793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/minicluster-data/master-0/data/instance:
uuid: "ff698dad44064a38b490adb40e61496e"
format_stamp: "Formatted at 2026-09-26 03:58:33 on dist-test-slave-nhpw"
server_key: "96969c97821dc706891341da6c7e44d3"
server_key_iv: "f9e450532e7f32dee172acb633ac1455"
server_key_version: "encryptionkey@0"
I20260926 03:58:33.074698 2793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/minicluster-data/master-0/wal/instance:
uuid: "ff698dad44064a38b490adb40e61496e"
format_stamp: "Formatted at 2026-09-26 03:58:33 on dist-test-slave-nhpw"
server_key: "96969c97821dc706891341da6c7e44d3"
server_key_iv: "f9e450532e7f32dee172acb633ac1455"
server_key_version: "encryptionkey@0"
I20260926 03:58:33.076086 2793 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 03:58:33.077005 2808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:33.077216 2793 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 03:58:33.077322 2793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/minicluster-data/master-0/wal
uuid: "ff698dad44064a38b490adb40e61496e"
format_stamp: "Formatted at 2026-09-26 03:58:33 on dist-test-slave-nhpw"
server_key: "96969c97821dc706891341da6c7e44d3"
server_key_iv: "f9e450532e7f32dee172acb633ac1455"
server_key_version: "encryptionkey@0"
I20260926 03:58:33.077404 2793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:58:33.090888 2793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:58:33.093323 2793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:58:33.093518 2793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:58:33.097836 2793 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40013
I20260926 03:58:33.097874 2860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40013 every 8 connection(s)
I20260926 03:58:33.098402 2793 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
I20260926 03:58:33.099092 2861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:58:33.101426 2861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff698dad44064a38b490adb40e61496e: Bootstrap starting.
I20260926 03:58:33.101970 2861 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff698dad44064a38b490adb40e61496e: Neither blocks nor log segments found. Creating new log.
I20260926 03:58:33.102200 2861 log.cc:824] T 00000000000000000000000000000000 P ff698dad44064a38b490adb40e61496e: Log is configured to *not* fsync() on all Append() calls
I20260926 03:58:33.102869 2861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff698dad44064a38b490adb40e61496e: No bootstrap required, opened a new log
I20260926 03:58:33.104336 2861 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff698dad44064a38b490adb40e61496e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff698dad44064a38b490adb40e61496e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40013 } }
I20260926 03:58:33.104473 2861 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff698dad44064a38b490adb40e61496e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:58:33.104496 2861 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff698dad44064a38b490adb40e61496e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff698dad44064a38b490adb40e61496e, State: Initialized, Role: FOLLOWER
I20260926 03:58:33.104631 2861 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff698dad44064a38b490adb40e61496e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff698dad44064a38b490adb40e61496e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40013 } }
I20260926 03:58:33.104741 2861 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff698dad44064a38b490adb40e61496e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:58:33.104765 2861 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff698dad44064a38b490adb40e61496e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:58:33.104838 2861 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff698dad44064a38b490adb40e61496e [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:58:33.104907 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 2793
I20260926 03:58:33.105044 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/minicluster-data/master-0/wal/instance
I20260926 03:58:33.105211 392 external_mini_cluster.cc:1428] Setting key bcbcb6bda837ed2ca3396bf046546ef9
I20260926 03:58:33.106068 2861 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff698dad44064a38b490adb40e61496e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff698dad44064a38b490adb40e61496e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40013 } }
I20260926 03:58:33.106177 2861 leader_election.cc:302] T 00000000000000000000000000000000 P ff698dad44064a38b490adb40e61496e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff698dad44064a38b490adb40e61496e; no voters:
I20260926 03:58:33.106325 2861 leader_election.cc:288] T 00000000000000000000000000000000 P ff698dad44064a38b490adb40e61496e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:58:33.106392 2866 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff698dad44064a38b490adb40e61496e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:58:33.106540 2866 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff698dad44064a38b490adb40e61496e [term 1 LEADER]: Becoming Leader. State: Replica: ff698dad44064a38b490adb40e61496e, State: Running, Role: LEADER
I20260926 03:58:33.106581 2861 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff698dad44064a38b490adb40e61496e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:58:33.106624 2866 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff698dad44064a38b490adb40e61496e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff698dad44064a38b490adb40e61496e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40013 } }
I20260926 03:58:33.106928 2866 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff698dad44064a38b490adb40e61496e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff698dad44064a38b490adb40e61496e. Latest consensus state: current_term: 1 leader_uuid: "ff698dad44064a38b490adb40e61496e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff698dad44064a38b490adb40e61496e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40013 } } }
I20260926 03:58:33.107021 2866 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff698dad44064a38b490adb40e61496e [sys.catalog]: This master's current role is: LEADER
I20260926 03:58:33.106938 2867 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff698dad44064a38b490adb40e61496e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff698dad44064a38b490adb40e61496e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff698dad44064a38b490adb40e61496e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40013 } } }
I20260926 03:58:33.107155 2867 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff698dad44064a38b490adb40e61496e [sys.catalog]: This master's current role is: LEADER
I20260926 03:58:33.107244 2872 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:58:33.107721 2872 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:58:33.109302 2872 catalog_manager.cc:1347] Generated new cluster ID: 02977463cb2548bb8d35203f536fdaf7
I20260926 03:58:33.109357 2872 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 03:58:33 dist-test-slave-nhpw krb5kdc[2780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395112, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 03:58:33.118389 2865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:33.105794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49582 (local address 127.0.98.62:40013)
0926 03:58:33.106043 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:33.106053 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:33.106089 (+ 36us) server_negotiation.cc:407] Connection header received
0926 03:58:33.106203 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:33.106211 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:33.106269 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:33.106438 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:33.107395 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:33.108993 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:33.110114 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:33.110743 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:33.110860 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:33.113702 (+ 2842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:33.113707 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:33.113707 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:33.113721 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:33.116590 (+ 2869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:33.117043 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:33.117046 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:33.117046 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:33.117164 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:33.117394 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:33.117395 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:33.117395 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:33.117721 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:33.117879 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:33.117998 (+ 119us) server_negotiation.cc:299] Negotiation successful
0926 03:58:33.118140 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":141,"threads_started":1}
I20260926 03:58:33.120891 2872 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:58:33.121615 2872 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:58:33.126287 2872 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff698dad44064a38b490adb40e61496e: Generated new TSK 0
I20260926 03:58:33.126502 2872 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 03:58:33.153694 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40013
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:58:33.234628 2887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:58:33.234840 2887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:58:33.234863 2887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:58:33.236100 2887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:58:33.236148 2887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:58:33.236176 2887 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 03:58:33.237746 2887 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40013
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2887
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:58:33.238020 2887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:58:33.238272 2887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:58:33.243336 2892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:33.243441 2893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:33.243497 2887 server_base.cc:1034] running on GCE node
W20260926 03:58:33.243512 2895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:33.243901 2887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:58:33.244194 2887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:58:33.245360 2887 hybrid_clock.cc:648] HybridClock initialized: now 1790395113245355 us; error 38 us; skew 500 ppm
Sep 26 03:58:33 dist-test-slave-nhpw krb5kdc[2780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395113, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:58:33.247910 2887 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 03:58:33.248292 2887 webserver.cc:460] Webserver started at http://127.0.98.1:43903/ using document root <none> and password file <none>
I20260926 03:58:33.248543 2887 fs_manager.cc:359] Metadata directory not provided
I20260926 03:58:33.248613 2887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:58:33.248780 2887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:58:33.249825 2887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-0/data/instance:
uuid: "d368cc8343d84f37ab39d7646d654091"
format_stamp: "Formatted at 2026-09-26 03:58:33 on dist-test-slave-nhpw"
server_key: "5e7e3eb47f3c19aff56a57d99a926df9"
server_key_iv: "4c5df1a2d14c1e0df41642220b5802b9"
server_key_version: "encryptionkey@0"
I20260926 03:58:33.250188 2887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d368cc8343d84f37ab39d7646d654091"
format_stamp: "Formatted at 2026-09-26 03:58:33 on dist-test-slave-nhpw"
server_key: "5e7e3eb47f3c19aff56a57d99a926df9"
server_key_iv: "4c5df1a2d14c1e0df41642220b5802b9"
server_key_version: "encryptionkey@0"
I20260926 03:58:33.251523 2887 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 03:58:33.252274 2902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:33.252555 2887 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 03:58:33.252674 2887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-0/wal
uuid: "d368cc8343d84f37ab39d7646d654091"
format_stamp: "Formatted at 2026-09-26 03:58:33 on dist-test-slave-nhpw"
server_key: "5e7e3eb47f3c19aff56a57d99a926df9"
server_key_iv: "4c5df1a2d14c1e0df41642220b5802b9"
server_key_version: "encryptionkey@0"
I20260926 03:58:33.252903 2887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:58:33.264894 2887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:58:33.267197 2887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:58:33.267366 2887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:58:33.267594 2887 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:58:33.267910 2887 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:58:33.267964 2887 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:33.267999 2887 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:58:33.268026 2887 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:33.274344 2887 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41925
I20260926 03:58:33.274436 3015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41925 every 8 connection(s)
I20260926 03:58:33.274767 2887 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
I20260926 03:58:33.278964 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 2887
I20260926 03:58:33.279089 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-0/wal/instance
I20260926 03:58:33.279276 392 external_mini_cluster.cc:1428] Setting key 7454149e55163385df407df3b0b847d3
Sep 26 03:58:33 dist-test-slave-nhpw krb5kdc[2780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395113, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 03:58:33.285030 2865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:33.275509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55799 (local address 127.0.98.62:40013)
0926 03:58:33.275568 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:33.275570 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:33.275785 (+ 215us) server_negotiation.cc:407] Connection header received
0926 03:58:33.276792 (+ 1007us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:33.276793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:33.276807 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:33.276860 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:33.277422 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:33.277910 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:33.279033 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:33.279354 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:33.279446 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:33.282079 (+ 2633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:33.282080 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:33.282080 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:33.282086 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:33.283544 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:33.284012 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:33.284013 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:33.284013 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:33.284073 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:33.284369 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:33.284370 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:33.284370 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:33.284474 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:33.284538 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:33.284902 (+ 364us) server_negotiation.cc:299] Negotiation successful
0926 03:58:33.284965 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 03:58:33.285337 3018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:33.275490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40013 (local address 127.0.98.1:55799)
0926 03:58:33.275639 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:33.275672 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:33.276689 (+ 1017us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:33.276912 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:33.276915 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:33.277164 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:33.277351 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:33.277354 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:33.277988 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:33.277989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:33.278931 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:33.278934 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:33.279459 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:33.279460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:33.279600 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:33.280193 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:33.280203 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:33.281983 (+ 1780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:58:33.283624 (+ 1641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:33.283630 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:33.283633 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:33.283936 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:33.284141 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:33.284142 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:33.284143 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:33.284290 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:33.284542 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:58:33.284549 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:58:33.284756 (+ 207us) client_negotiation.cc:769] Sending connection context
0926 03:58:33.284886 (+ 130us) client_negotiation.cc:240] Negotiation successful
0926 03:58:33.285066 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":87,"threads_started":1}
I20260926 03:58:33.285923 3016 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40013
I20260926 03:58:33.286049 3016 heartbeater.cc:464] Registering TS with master...
I20260926 03:58:33.286228 3016 heartbeater.cc:511] Master 127.0.98.62:40013 requested a full tablet report, sending...
I20260926 03:58:33.286731 2825 ts_manager.cc:194] Registered new tserver with Master: d368cc8343d84f37ab39d7646d654091 (127.0.98.1:41925)
I20260926 03:58:33.287336 2825 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55799
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 03:58:33.306516 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40013
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:58:33.387286 3021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:58:33.387480 3021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:58:33.387502 3021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:58:33.388568 3021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:58:33.388605 3021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:58:33.388645 3021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 03:58:33.390040 3021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40013
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:58:33.390215 3021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:58:33.390384 3021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:58:33.395473 3027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:33.395593 3029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:33.395475 3026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:33.395747 3021 server_base.cc:1034] running on GCE node
I20260926 03:58:33.395960 3021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:58:33.396165 3021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:58:33.397351 3021 hybrid_clock.cc:648] HybridClock initialized: now 1790395113397330 us; error 36 us; skew 500 ppm
Sep 26 03:58:33 dist-test-slave-nhpw krb5kdc[2780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395113, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:58:33.399901 3021 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 03:58:33.400403 3021 webserver.cc:460] Webserver started at http://127.0.98.2:36735/ using document root <none> and password file <none>
I20260926 03:58:33.400688 3021 fs_manager.cc:359] Metadata directory not provided
I20260926 03:58:33.400768 3021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:58:33.400904 3021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:58:33.401875 3021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-1/data/instance:
uuid: "f47cd45491864f2a8804a09a3d910c6f"
format_stamp: "Formatted at 2026-09-26 03:58:33 on dist-test-slave-nhpw"
server_key: "82c5be954274f000d619a682dd5a19bd"
server_key_iv: "1d6b9647631ba84cfab00391caa9fde1"
server_key_version: "encryptionkey@0"
I20260926 03:58:33.402175 3021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f47cd45491864f2a8804a09a3d910c6f"
format_stamp: "Formatted at 2026-09-26 03:58:33 on dist-test-slave-nhpw"
server_key: "82c5be954274f000d619a682dd5a19bd"
server_key_iv: "1d6b9647631ba84cfab00391caa9fde1"
server_key_version: "encryptionkey@0"
I20260926 03:58:33.403390 3021 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 03:58:33.404057 3036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:33.404256 3021 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 03:58:33.404336 3021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-1/wal
uuid: "f47cd45491864f2a8804a09a3d910c6f"
format_stamp: "Formatted at 2026-09-26 03:58:33 on dist-test-slave-nhpw"
server_key: "82c5be954274f000d619a682dd5a19bd"
server_key_iv: "1d6b9647631ba84cfab00391caa9fde1"
server_key_version: "encryptionkey@0"
I20260926 03:58:33.404387 3021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:58:33.431483 3021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:58:33.433634 3021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:58:33.433745 3021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:58:33.433933 3021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:58:33.434249 3021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:58:33.434283 3021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:33.434305 3021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:58:33.434353 3021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:33.440001 3021 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35549
I20260926 03:58:33.440042 3149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35549 every 8 connection(s)
I20260926 03:58:33.440346 3021 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
I20260926 03:58:33.441416 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 3021
I20260926 03:58:33.441533 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-1/wal/instance
I20260926 03:58:33.441715 392 external_mini_cluster.cc:1428] Setting key a8ef94bf685eda2afc338ca8f7703397
Sep 26 03:58:33 dist-test-slave-nhpw krb5kdc[2780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395113, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 03:58:33.450752 2865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:33.441069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41547 (local address 127.0.98.62:40013)
0926 03:58:33.441112 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:33.441114 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:33.441334 (+ 220us) server_negotiation.cc:407] Connection header received
0926 03:58:33.442522 (+ 1188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:33.442524 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:33.442537 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:33.442575 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:33.443209 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:33.444013 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:33.445085 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:33.445421 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:33.445500 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:33.447670 (+ 2170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:33.447672 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:33.447673 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:33.447681 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:33.449371 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:33.449783 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:33.449785 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:33.449786 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:33.449847 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:33.450102 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:33.450103 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:33.450104 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:33.450223 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:33.450302 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:33.450573 (+ 271us) server_negotiation.cc:299] Negotiation successful
0926 03:58:33.450657 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 03:58:33.450989 3152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:33.441048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40013 (local address 127.0.98.2:41547)
0926 03:58:33.441218 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:33.441252 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:33.442422 (+ 1170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:33.442643 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:33.442647 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:33.442912 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:33.443105 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:33.443110 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:33.444113 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:33.444114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:33.445002 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:33.445005 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:33.445528 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:33.445529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:33.445667 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:33.446228 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:33.446236 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:33.447576 (+ 1340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:58:33.449453 (+ 1877us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:33.449459 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:33.449461 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:33.449675 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:33.449915 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:33.449916 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:33.449917 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:33.450030 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:33.450290 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:58:33.450296 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:58:33.450436 (+ 140us) client_negotiation.cc:769] Sending connection context
0926 03:58:33.450553 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 03:58:33.450714 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":75,"threads_started":1}
I20260926 03:58:33.451432 3150 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40013
I20260926 03:58:33.451556 3150 heartbeater.cc:464] Registering TS with master...
I20260926 03:58:33.451728 3150 heartbeater.cc:511] Master 127.0.98.62:40013 requested a full tablet report, sending...
I20260926 03:58:33.452093 2825 ts_manager.cc:194] Registered new tserver with Master: f47cd45491864f2a8804a09a3d910c6f (127.0.98.2:35549)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260926 03:58:33.452538 2825 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41547
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 03:58:33.467371 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40013
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:58:33.547782 3155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:58:33.548002 3155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:58:33.548038 3155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:58:33.549247 3155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:58:33.549295 3155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:58:33.549322 3155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 03:58:33.550829 3155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40013
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:58:33.551080 3155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:58:33.551316 3155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:58:33.556314 3163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:33.556416 3161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:33.556341 3160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:33.556550 3155 server_base.cc:1034] running on GCE node
I20260926 03:58:33.556798 3155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:58:33.556986 3155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:58:33.558130 3155 hybrid_clock.cc:648] HybridClock initialized: now 1790395113558118 us; error 29 us; skew 500 ppm
Sep 26 03:58:33 dist-test-slave-nhpw krb5kdc[2780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395113, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:58:33.560489 3155 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 03:58:33.561002 3155 webserver.cc:460] Webserver started at http://127.0.98.3:38019/ using document root <none> and password file <none>
I20260926 03:58:33.561264 3155 fs_manager.cc:359] Metadata directory not provided
I20260926 03:58:33.561331 3155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:58:33.561482 3155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:58:33.562498 3155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-2/data/instance:
uuid: "cfeb3224ae1448ea82fe011aca9a02cc"
format_stamp: "Formatted at 2026-09-26 03:58:33 on dist-test-slave-nhpw"
server_key: "bf7594fe4138a2dc752e78181371e77b"
server_key_iv: "633924ba9c7aedea4995e82e8b23098f"
server_key_version: "encryptionkey@0"
I20260926 03:58:33.562860 3155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-2/wal/instance:
uuid: "cfeb3224ae1448ea82fe011aca9a02cc"
format_stamp: "Formatted at 2026-09-26 03:58:33 on dist-test-slave-nhpw"
server_key: "bf7594fe4138a2dc752e78181371e77b"
server_key_iv: "633924ba9c7aedea4995e82e8b23098f"
server_key_version: "encryptionkey@0"
I20260926 03:58:33.564179 3155 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 03:58:33.564939 3170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:33.565126 3155 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 03:58:33.565220 3155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-2/wal
uuid: "cfeb3224ae1448ea82fe011aca9a02cc"
format_stamp: "Formatted at 2026-09-26 03:58:33 on dist-test-slave-nhpw"
server_key: "bf7594fe4138a2dc752e78181371e77b"
server_key_iv: "633924ba9c7aedea4995e82e8b23098f"
server_key_version: "encryptionkey@0"
I20260926 03:58:33.565300 3155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:58:33.581473 3155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:58:33.583467 3155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:58:33.583659 3155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:58:33.583918 3155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:58:33.584224 3155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:58:33.584278 3155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:33.584317 3155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:58:33.584347 3155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:33.590024 3155 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38643
I20260926 03:58:33.590065 3283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38643 every 8 connection(s)
I20260926 03:58:33.590370 3155 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
I20260926 03:58:33.591784 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 3155
I20260926 03:58:33.591881 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-2/wal/instance
I20260926 03:58:33.592061 392 external_mini_cluster.cc:1428] Setting key 955fbed46b1288f65f045232395bcd51
Sep 26 03:58:33 dist-test-slave-nhpw krb5kdc[2780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395113, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 03:58:33.599576 2865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:33.591044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44365 (local address 127.0.98.62:40013)
0926 03:58:33.591117 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:33.591119 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:33.591331 (+ 212us) server_negotiation.cc:407] Connection header received
0926 03:58:33.592391 (+ 1060us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:33.592392 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:33.592405 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:33.592466 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:33.593255 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:33.593782 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:33.594785 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:33.594972 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:33.595037 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:33.597169 (+ 2132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:33.597170 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:33.597170 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:33.597177 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:33.598443 (+ 1266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:33.598804 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:33.598805 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:33.598805 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:33.598852 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:33.599073 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:33.599073 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:33.599073 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:33.599174 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:33.599224 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:33.599448 (+ 224us) server_negotiation.cc:299] Negotiation successful
0926 03:58:33.599505 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 03:58:33.599741 3286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:33.591035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40013 (local address 127.0.98.3:44365)
0926 03:58:33.591208 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:33.591242 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:33.592259 (+ 1017us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:33.592534 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:33.592544 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:33.592961 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:33.593157 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:33.593161 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:33.593867 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:33.593868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:33.594692 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:33.594694 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:33.595065 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:33.595066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:33.595162 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:33.595681 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:33.595696 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:33.597089 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:58:33.598521 (+ 1432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:33.598528 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:33.598530 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:33.598740 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:33.598908 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:33.598909 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:33.598909 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:33.599003 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:33.599230 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:58:33.599234 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:58:33.599341 (+ 107us) client_negotiation.cc:769] Sending connection context
0926 03:58:33.599423 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 03:58:33.599538 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":96,"threads_started":1}
I20260926 03:58:33.600209 3284 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40013
I20260926 03:58:33.600308 3284 heartbeater.cc:464] Registering TS with master...
I20260926 03:58:33.600476 3284 heartbeater.cc:511] Master 127.0.98.62:40013 requested a full tablet report, sending...
I20260926 03:58:33.600875 2825 ts_manager.cc:194] Registered new tserver with Master: cfeb3224ae1448ea82fe011aca9a02cc (127.0.98.3:38643)
I20260926 03:58:33.601243 2825 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44365
I20260926 03:58:33.603711 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 03:58:33.614567 2865 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:58:33.613710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49592 (local address 127.0.98.62:40013)
0926 03:58:33.613769 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:33.613771 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:33.613896 (+ 125us) server_negotiation.cc:407] Connection header received
0926 03:58:33.613941 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:33.613942 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:33.613952 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:33.613998 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:33.614474 (+ 476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49592: BlockingRecv error: recv got EOF from 127.0.0.1:49592 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 03:58:33.615402 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 2887
I20260926 03:58:33.620234 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 3021
I20260926 03:58:33.624372 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 3155
I20260926 03:58:33.628551 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 2793
2026-09-26T03:58:33Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1816 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790395095474643-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:58:33 dist-test-slave-nhpw krb5kdc[3296](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:58:33 dist-test-slave-nhpw krb5kdc[3296](info): set up 2 sockets
Sep 26 03:58:33 dist-test-slave-nhpw krb5kdc[3296](info): commencing operation
krb5kdc: starting...
W20260926 03:58:34.676501 392 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790395095474643-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790395095474643-392-0/krb5kdc/test-user.keytab.
Sep 26 03:58:34 dist-test-slave-nhpw krb5kdc[3296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395114, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:58:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:58:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 03:58:34.795565 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790395095474643-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790395095474643-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790395095474643-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44949
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44949
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:58:34.873358 3309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:58:34.873554 3309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:58:34.873575 3309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:58:34.874641 3309 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:58:34.874675 3309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:58:34.874688 3309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:58:34.874701 3309 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:58:34.874712 3309 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:58:34.876124 3309 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790395095474643-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790395095474643-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44949
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790395095474643-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:58:34.876304 3309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:58:34.876466 3309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:58:34.881589 3315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:34.882658 3309 server_base.cc:1034] running on GCE node
W20260926 03:58:34.883021 3314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:34.883823 3317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:34.884160 3309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:58:34.884493 3309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:58:34.885677 3309 hybrid_clock.cc:648] HybridClock initialized: now 1790395114885666 us; error 53 us; skew 500 ppm
Sep 26 03:58:34 dist-test-slave-nhpw krb5kdc[3296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395114, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:58:34.893651 3309 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 03:58:34.894188 3309 webserver.cc:460] Webserver started at http://127.0.98.62:37225/ using document root <none> and password file <none>
I20260926 03:58:34.894438 3309 fs_manager.cc:359] Metadata directory not provided
I20260926 03:58:34.894510 3309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:58:34.894656 3309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:58:34.895725 3309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790395095474643-392-0/minicluster-data/master-0/data/instance:
uuid: "30b03cdfed6548348508bd321f833adb"
format_stamp: "Formatted at 2026-09-26 03:58:34 on dist-test-slave-nhpw"
server_key: "43a9697767a07939196a5f3ce2f1a5a7"
server_key_iv: "c7f1f13e015e87d6c62a2ef1865fb01a"
server_key_version: "encryptionkey@0"
I20260926 03:58:34.896114 3309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790395095474643-392-0/minicluster-data/master-0/wal/instance:
uuid: "30b03cdfed6548348508bd321f833adb"
format_stamp: "Formatted at 2026-09-26 03:58:34 on dist-test-slave-nhpw"
server_key: "43a9697767a07939196a5f3ce2f1a5a7"
server_key_iv: "c7f1f13e015e87d6c62a2ef1865fb01a"
server_key_version: "encryptionkey@0"
I20260926 03:58:34.897598 3309 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 03:58:34.898362 3324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:34.898557 3309 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 03:58:34.898635 3309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790395095474643-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790395095474643-392-0/minicluster-data/master-0/wal
uuid: "30b03cdfed6548348508bd321f833adb"
format_stamp: "Formatted at 2026-09-26 03:58:34 on dist-test-slave-nhpw"
server_key: "43a9697767a07939196a5f3ce2f1a5a7"
server_key_iv: "c7f1f13e015e87d6c62a2ef1865fb01a"
server_key_version: "encryptionkey@0"
I20260926 03:58:34.898712 3309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790395095474643-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790395095474643-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790395095474643-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:58:34.914979 3309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:58:34.917341 3309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:58:34.917506 3309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:58:34.921432 3309 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44949
I20260926 03:58:34.921464 3376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44949 every 8 connection(s)
I20260926 03:58:34.921828 3309 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
I20260926 03:58:34.922869 3377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:58:34.925805 3377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30b03cdfed6548348508bd321f833adb: Bootstrap starting.
I20260926 03:58:34.926400 3377 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30b03cdfed6548348508bd321f833adb: Neither blocks nor log segments found. Creating new log.
I20260926 03:58:34.926689 3377 log.cc:824] T 00000000000000000000000000000000 P 30b03cdfed6548348508bd321f833adb: Log is configured to *not* fsync() on all Append() calls
I20260926 03:58:34.927363 3377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30b03cdfed6548348508bd321f833adb: No bootstrap required, opened a new log
I20260926 03:58:34.928960 3377 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30b03cdfed6548348508bd321f833adb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30b03cdfed6548348508bd321f833adb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44949 } }
I20260926 03:58:34.929137 3377 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30b03cdfed6548348508bd321f833adb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:58:34.929185 3377 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30b03cdfed6548348508bd321f833adb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30b03cdfed6548348508bd321f833adb, State: Initialized, Role: FOLLOWER
I20260926 03:58:34.929365 3377 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30b03cdfed6548348508bd321f833adb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30b03cdfed6548348508bd321f833adb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44949 } }
I20260926 03:58:34.929466 3377 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30b03cdfed6548348508bd321f833adb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:58:34.929507 3377 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30b03cdfed6548348508bd321f833adb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:58:34.929554 3377 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30b03cdfed6548348508bd321f833adb [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:58:34.930023 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 3309
I20260926 03:58:34.930094 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790395095474643-392-0/minicluster-data/master-0/wal/instance
I20260926 03:58:34.930263 392 external_mini_cluster.cc:1428] Setting key 6983435d4d8a531333407516c8db8f8d
I20260926 03:58:34.930445 3377 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30b03cdfed6548348508bd321f833adb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30b03cdfed6548348508bd321f833adb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44949 } }
I20260926 03:58:34.930555 3377 leader_election.cc:302] T 00000000000000000000000000000000 P 30b03cdfed6548348508bd321f833adb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30b03cdfed6548348508bd321f833adb; no voters:
I20260926 03:58:34.930712 3377 leader_election.cc:288] T 00000000000000000000000000000000 P 30b03cdfed6548348508bd321f833adb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:58:34.930742 3380 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30b03cdfed6548348508bd321f833adb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:58:34.930948 3380 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30b03cdfed6548348508bd321f833adb [term 1 LEADER]: Becoming Leader. State: Replica: 30b03cdfed6548348508bd321f833adb, State: Running, Role: LEADER
I20260926 03:58:34.931057 3377 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30b03cdfed6548348508bd321f833adb [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:58:34.931072 3380 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30b03cdfed6548348508bd321f833adb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30b03cdfed6548348508bd321f833adb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44949 } }
I20260926 03:58:34.931571 3383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30b03cdfed6548348508bd321f833adb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "30b03cdfed6548348508bd321f833adb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30b03cdfed6548348508bd321f833adb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44949 } } }
I20260926 03:58:34.931665 3383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30b03cdfed6548348508bd321f833adb [sys.catalog]: This master's current role is: LEADER
I20260926 03:58:34.931857 3391 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:58:34.931562 3384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30b03cdfed6548348508bd321f833adb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30b03cdfed6548348508bd321f833adb. Latest consensus state: current_term: 1 leader_uuid: "30b03cdfed6548348508bd321f833adb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30b03cdfed6548348508bd321f833adb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44949 } } }
I20260926 03:58:34.932509 3384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30b03cdfed6548348508bd321f833adb [sys.catalog]: This master's current role is: LEADER
I20260926 03:58:34.932423 3391 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:58:34.934266 3391 catalog_manager.cc:1347] Generated new cluster ID: c2f260f4ed55435b81b14d4f5e12988b
I20260926 03:58:34.934319 3391 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 03:58:34 dist-test-slave-nhpw krb5kdc[3296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395114, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 03:58:34.942477 3382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:34.930740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33762 (local address 127.0.98.62:44949)
0926 03:58:34.931146 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:34.931167 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:34.931196 (+ 29us) server_negotiation.cc:407] Connection header received
0926 03:58:34.931314 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:34.931319 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:34.931368 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:34.932608 (+ 1240us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:34.932796 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:34.934161 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:34.935284 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:34.935903 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:34.936003 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:34.938793 (+ 2790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:34.938798 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:34.938798 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:34.938811 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:34.940899 (+ 2088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:34.941258 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:34.941261 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:34.941261 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:34.941375 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:34.941579 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:34.941580 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:34.941580 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:34.941901 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:34.942062 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:34.942132 (+ 70us) server_negotiation.cc:299] Negotiation successful
0926 03:58:34.942266 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":353,"thread_start_us":92,"threads_started":1}
I20260926 03:58:34.943306 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:58:34.957525 3391 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:58:34.958257 3391 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:58:34.959388 3382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:34.955270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33768 (local address 127.0.98.62:44949)
0926 03:58:34.955317 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:34.955319 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:34.955462 (+ 143us) server_negotiation.cc:407] Connection header received
0926 03:58:34.955522 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:34.955523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:34.955533 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:34.955585 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:34.956019 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:34.956776 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:34.957824 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:34.958123 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:34.958193 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:34.958899 (+ 706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:34.958901 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:34.958901 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:58:34.959118 (+ 217us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:34.959134 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:58:34.959141 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:34.959196 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:34.959276 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 03:58:34.959315 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 03:58:34.966015 3391 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 30b03cdfed6548348508bd321f833adb: Generated new TSK 0
I20260926 03:58:34.966220 3391 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:58:34.972219 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 3309
2026-09-26T03:58:34Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1341 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:58:35 dist-test-slave-nhpw krb5kdc[3410](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:58:35 dist-test-slave-nhpw krb5kdc[3410](info): set up 2 sockets
Sep 26 03:58:35 dist-test-slave-nhpw krb5kdc[3410](info): commencing operation
krb5kdc: starting...
W20260926 03:58:36.016538 392 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/krb5kdc/test-user.keytab.
Sep 26 03:58:36 dist-test-slave-nhpw krb5kdc[3410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:58:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:58:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 03:58:36.136575 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35345
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35345
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:58:36.221635 3423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:58:36.221855 3423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:58:36.221879 3423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:58:36.222963 3423 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:58:36.222999 3423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:58:36.223013 3423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:58:36.223026 3423 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:58:36.223038 3423 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:58:36.224413 3423 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35345
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3423
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:58:36.224607 3423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:58:36.224855 3423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:58:36.229568 3429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:36.229759 3431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:36.229596 3428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:36.229774 3423 server_base.cc:1034] running on GCE node
I20260926 03:58:36.230077 3423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:58:36.230315 3423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:58:36.231477 3423 hybrid_clock.cc:648] HybridClock initialized: now 1790395116231458 us; error 32 us; skew 500 ppm
Sep 26 03:58:36 dist-test-slave-nhpw krb5kdc[3410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395116, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:58:36.234031 3423 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 03:58:36.234457 3423 webserver.cc:460] Webserver started at http://127.0.98.62:33275/ using document root <none> and password file <none>
I20260926 03:58:36.234688 3423 fs_manager.cc:359] Metadata directory not provided
I20260926 03:58:36.234758 3423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:58:36.234908 3423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:58:36.235985 3423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/minicluster-data/master-0/data/instance:
uuid: "84f8d823d86c475fb65cf327699519f8"
format_stamp: "Formatted at 2026-09-26 03:58:36 on dist-test-slave-nhpw"
server_key: "87752b589cde95bf0e852faa9c7647ec"
server_key_iv: "c2ad9d25ff67510a33d338f3c908aef6"
server_key_version: "encryptionkey@0"
I20260926 03:58:36.236342 3423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/minicluster-data/master-0/wal/instance:
uuid: "84f8d823d86c475fb65cf327699519f8"
format_stamp: "Formatted at 2026-09-26 03:58:36 on dist-test-slave-nhpw"
server_key: "87752b589cde95bf0e852faa9c7647ec"
server_key_iv: "c2ad9d25ff67510a33d338f3c908aef6"
server_key_version: "encryptionkey@0"
I20260926 03:58:36.237695 3423 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 03:58:36.238425 3438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:36.238600 3423 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 03:58:36.238701 3423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/minicluster-data/master-0/wal
uuid: "84f8d823d86c475fb65cf327699519f8"
format_stamp: "Formatted at 2026-09-26 03:58:36 on dist-test-slave-nhpw"
server_key: "87752b589cde95bf0e852faa9c7647ec"
server_key_iv: "c2ad9d25ff67510a33d338f3c908aef6"
server_key_version: "encryptionkey@0"
I20260926 03:58:36.238783 3423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:58:36.250808 3423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:58:36.252681 3423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:58:36.252846 3423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:58:36.256592 3423 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35345
I20260926 03:58:36.256693 3490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35345 every 8 connection(s)
I20260926 03:58:36.257031 3423 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
I20260926 03:58:36.258041 3491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:58:36.260165 3491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84f8d823d86c475fb65cf327699519f8: Bootstrap starting.
I20260926 03:58:36.260766 3491 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84f8d823d86c475fb65cf327699519f8: Neither blocks nor log segments found. Creating new log.
I20260926 03:58:36.261001 3491 log.cc:824] T 00000000000000000000000000000000 P 84f8d823d86c475fb65cf327699519f8: Log is configured to *not* fsync() on all Append() calls
I20260926 03:58:36.261230 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 3423
I20260926 03:58:36.261327 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/minicluster-data/master-0/wal/instance
I20260926 03:58:36.261514 392 external_mini_cluster.cc:1428] Setting key ad5f0172b6f4bf9524af0580b65c6dc6
I20260926 03:58:36.261842 3491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84f8d823d86c475fb65cf327699519f8: No bootstrap required, opened a new log
I20260926 03:58:36.263453 3491 raft_consensus.cc:348] T 00000000000000000000000000000000 P 84f8d823d86c475fb65cf327699519f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84f8d823d86c475fb65cf327699519f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35345 } }
I20260926 03:58:36.263583 3491 raft_consensus.cc:374] T 00000000000000000000000000000000 P 84f8d823d86c475fb65cf327699519f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:58:36.263607 3491 raft_consensus.cc:729] T 00000000000000000000000000000000 P 84f8d823d86c475fb65cf327699519f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84f8d823d86c475fb65cf327699519f8, State: Initialized, Role: FOLLOWER
I20260926 03:58:36.263762 3491 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84f8d823d86c475fb65cf327699519f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84f8d823d86c475fb65cf327699519f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35345 } }
I20260926 03:58:36.263855 3491 raft_consensus.cc:388] T 00000000000000000000000000000000 P 84f8d823d86c475fb65cf327699519f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:58:36.263888 3491 raft_consensus.cc:482] T 00000000000000000000000000000000 P 84f8d823d86c475fb65cf327699519f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:58:36.263940 3491 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 84f8d823d86c475fb65cf327699519f8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:58:36.264954 3491 raft_consensus.cc:504] T 00000000000000000000000000000000 P 84f8d823d86c475fb65cf327699519f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84f8d823d86c475fb65cf327699519f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35345 } }
I20260926 03:58:36.265040 3491 leader_election.cc:302] T 00000000000000000000000000000000 P 84f8d823d86c475fb65cf327699519f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 84f8d823d86c475fb65cf327699519f8; no voters:
I20260926 03:58:36.265229 3491 leader_election.cc:288] T 00000000000000000000000000000000 P 84f8d823d86c475fb65cf327699519f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:58:36.265277 3496 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 84f8d823d86c475fb65cf327699519f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:58:36.265408 3496 raft_consensus.cc:686] T 00000000000000000000000000000000 P 84f8d823d86c475fb65cf327699519f8 [term 1 LEADER]: Becoming Leader. State: Replica: 84f8d823d86c475fb65cf327699519f8, State: Running, Role: LEADER
I20260926 03:58:36.265475 3491 sys_catalog.cc:564] T 00000000000000000000000000000000 P 84f8d823d86c475fb65cf327699519f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:58:36.265542 3496 consensus_queue.cc:237] T 00000000000000000000000000000000 P 84f8d823d86c475fb65cf327699519f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84f8d823d86c475fb65cf327699519f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35345 } }
I20260926 03:58:36.266022 3497 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84f8d823d86c475fb65cf327699519f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "84f8d823d86c475fb65cf327699519f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84f8d823d86c475fb65cf327699519f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35345 } } }
I20260926 03:58:36.266111 3497 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84f8d823d86c475fb65cf327699519f8 [sys.catalog]: This master's current role is: LEADER
I20260926 03:58:36.266340 3498 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84f8d823d86c475fb65cf327699519f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 84f8d823d86c475fb65cf327699519f8. Latest consensus state: current_term: 1 leader_uuid: "84f8d823d86c475fb65cf327699519f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "84f8d823d86c475fb65cf327699519f8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35345 } } }
I20260926 03:58:36.266454 3498 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84f8d823d86c475fb65cf327699519f8 [sys.catalog]: This master's current role is: LEADER
I20260926 03:58:36.266633 3505 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:58:36.267139 3505 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:58:36.268589 3505 catalog_manager.cc:1347] Generated new cluster ID: e7941f9bfe3c4965a6d5ae885b654e63
I20260926 03:58:36.268667 3505 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 03:58:36 dist-test-slave-nhpw krb5kdc[3410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395116, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 03:58:36.274248 3495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:36.261982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48224 (local address 127.0.98.62:35345)
0926 03:58:36.262148 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:36.262158 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:36.262191 (+ 33us) server_negotiation.cc:407] Connection header received
0926 03:58:36.263469 (+ 1278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:36.263476 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:36.263531 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:36.263698 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:36.264478 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:36.265758 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:36.266987 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:36.267470 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:36.267604 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:36.270423 (+ 2819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:36.270427 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:36.270427 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:36.270439 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:36.272411 (+ 1972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:36.272821 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:36.272824 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:36.272825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:36.272960 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:36.273164 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:36.273165 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:36.273165 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:36.273490 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:36.273651 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:36.273726 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 03:58:36.273867 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":73,"threads_started":1}
I20260926 03:58:36.292706 3505 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:58:36.293488 3505 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:58:36.302685 3505 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 84f8d823d86c475fb65cf327699519f8: Generated new TSK 0
I20260926 03:58:36.302899 3505 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 03:58:36.335505 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35345
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:58:36.417842 3517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:58:36.418107 3517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:58:36.418146 3517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:58:36.419364 3517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:58:36.419440 3517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:58:36.419486 3517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 03:58:36.421047 3517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35345
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:58:36.421317 3517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:58:36.421559 3517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:58:36.426939 3525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:36.427026 3523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:36.427052 3522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:36.427244 3517 server_base.cc:1034] running on GCE node
I20260926 03:58:36.427605 3517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:58:36.428038 3517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:58:36.429231 3517 hybrid_clock.cc:648] HybridClock initialized: now 1790395116429217 us; error 47 us; skew 500 ppm
Sep 26 03:58:36 dist-test-slave-nhpw krb5kdc[3410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395116, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:58:36.431993 3517 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 03:58:36.432488 3517 webserver.cc:460] Webserver started at http://127.0.98.1:41807/ using document root <none> and password file <none>
I20260926 03:58:36.432747 3517 fs_manager.cc:359] Metadata directory not provided
I20260926 03:58:36.432793 3517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:58:36.432885 3517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:58:36.433895 3517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-0/data/instance:
uuid: "c678d8fb1ee44e7c89e65aafb5f50220"
format_stamp: "Formatted at 2026-09-26 03:58:36 on dist-test-slave-nhpw"
server_key: "5d85de9c1e0713741d4d853087c0396d"
server_key_iv: "4d62f1b254ff707ae86237b1908a8019"
server_key_version: "encryptionkey@0"
I20260926 03:58:36.434207 3517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c678d8fb1ee44e7c89e65aafb5f50220"
format_stamp: "Formatted at 2026-09-26 03:58:36 on dist-test-slave-nhpw"
server_key: "5d85de9c1e0713741d4d853087c0396d"
server_key_iv: "4d62f1b254ff707ae86237b1908a8019"
server_key_version: "encryptionkey@0"
I20260926 03:58:36.435500 3517 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 03:58:36.436273 3532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:36.436542 3517 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 03:58:36.436643 3517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-0/wal
uuid: "c678d8fb1ee44e7c89e65aafb5f50220"
format_stamp: "Formatted at 2026-09-26 03:58:36 on dist-test-slave-nhpw"
server_key: "5d85de9c1e0713741d4d853087c0396d"
server_key_iv: "4d62f1b254ff707ae86237b1908a8019"
server_key_version: "encryptionkey@0"
I20260926 03:58:36.436753 3517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:58:36.446679 3517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:58:36.449083 3517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:58:36.449227 3517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:58:36.449467 3517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:58:36.449781 3517 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:58:36.449819 3517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:36.449841 3517 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:58:36.449888 3517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:36.455722 3517 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35245
I20260926 03:58:36.455798 3645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35245 every 8 connection(s)
I20260926 03:58:36.456110 3517 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
I20260926 03:58:36.460757 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 3517
I20260926 03:58:36.460880 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-0/wal/instance
I20260926 03:58:36.461082 392 external_mini_cluster.cc:1428] Setting key 77aff4b6342d395e3767af1aadea1347
Sep 26 03:58:36 dist-test-slave-nhpw krb5kdc[3410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395116, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 03:58:36.466045 3495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:36.456836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33713 (local address 127.0.98.62:35345)
0926 03:58:36.456895 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:36.456897 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:36.457080 (+ 183us) server_negotiation.cc:407] Connection header received
0926 03:58:36.457619 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:36.457621 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:36.457637 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:36.457693 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:36.458639 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:36.459154 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:36.460156 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:36.460378 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:36.460470 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:36.462886 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:36.462888 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:36.462888 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:36.462898 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:36.464362 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:36.464974 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:36.464975 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:36.464975 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:36.465052 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:36.465369 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:36.465370 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:36.465371 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:36.465492 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:36.465568 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:36.465844 (+ 276us) server_negotiation.cc:299] Negotiation successful
0926 03:58:36.465939 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 03:58:36.466369 3648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:36.456798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35345 (local address 127.0.98.1:33713)
0926 03:58:36.456932 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:36.456977 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:36.457512 (+ 535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:36.457760 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:36.457764 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:36.458325 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:36.458519 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:36.458524 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:36.459249 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:36.459250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:36.460058 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:36.460060 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:36.460480 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:36.460482 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:36.460628 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:36.461203 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:36.461211 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:36.462786 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:58:36.464481 (+ 1695us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:36.464488 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:36.464491 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:36.464805 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:36.465127 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:36.465128 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:36.465128 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:36.465247 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:36.465559 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:58:36.465566 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:58:36.465711 (+ 145us) client_negotiation.cc:769] Sending connection context
0926 03:58:36.465822 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 03:58:36.466026 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":62,"threads_started":1}
I20260926 03:58:36.466923 3646 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35345
I20260926 03:58:36.467037 3646 heartbeater.cc:464] Registering TS with master...
I20260926 03:58:36.467259 3646 heartbeater.cc:511] Master 127.0.98.62:35345 requested a full tablet report, sending...
I20260926 03:58:36.467826 3455 ts_manager.cc:194] Registered new tserver with Master: c678d8fb1ee44e7c89e65aafb5f50220 (127.0.98.1:35245)
I20260926 03:58:36.468828 3455 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33713
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 03:58:36.489176 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35345
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:58:36.572539 3651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:58:36.572819 3651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:58:36.572861 3651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:58:36.574131 3651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:58:36.574213 3651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:58:36.574258 3651 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 03:58:36.575773 3651 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35345
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3651
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:58:36.576064 3651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:58:36.576308 3651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:58:36.581233 3656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:36.581274 3659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:36.581241 3657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:36.581387 3651 server_base.cc:1034] running on GCE node
I20260926 03:58:36.581630 3651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:58:36.581871 3651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:58:36.583041 3651 hybrid_clock.cc:648] HybridClock initialized: now 1790395116583025 us; error 49 us; skew 500 ppm
Sep 26 03:58:36 dist-test-slave-nhpw krb5kdc[3410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395116, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:58:36.585593 3651 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 03:58:36.586069 3651 webserver.cc:460] Webserver started at http://127.0.98.2:40035/ using document root <none> and password file <none>
I20260926 03:58:36.586290 3651 fs_manager.cc:359] Metadata directory not provided
I20260926 03:58:36.586354 3651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:58:36.586500 3651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:58:36.587524 3651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-1/data/instance:
uuid: "b39edf9305c043e4b14c77e63550430c"
format_stamp: "Formatted at 2026-09-26 03:58:36 on dist-test-slave-nhpw"
server_key: "bc22c5413cb0328f06e8179afd82e288"
server_key_iv: "f8ea4e740a71548f8585e09c54a374af"
server_key_version: "encryptionkey@0"
I20260926 03:58:36.587903 3651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b39edf9305c043e4b14c77e63550430c"
format_stamp: "Formatted at 2026-09-26 03:58:36 on dist-test-slave-nhpw"
server_key: "bc22c5413cb0328f06e8179afd82e288"
server_key_iv: "f8ea4e740a71548f8585e09c54a374af"
server_key_version: "encryptionkey@0"
I20260926 03:58:36.589329 3651 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 03:58:36.590137 3666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:36.590319 3651 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 03:58:36.590417 3651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-1/wal
uuid: "b39edf9305c043e4b14c77e63550430c"
format_stamp: "Formatted at 2026-09-26 03:58:36 on dist-test-slave-nhpw"
server_key: "bc22c5413cb0328f06e8179afd82e288"
server_key_iv: "f8ea4e740a71548f8585e09c54a374af"
server_key_version: "encryptionkey@0"
I20260926 03:58:36.590488 3651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:58:36.599071 3651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:58:36.601358 3651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:58:36.601521 3651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:58:36.601751 3651 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:58:36.602057 3651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:58:36.602108 3651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:36.602149 3651 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:58:36.602176 3651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:36.608017 3651 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39287
I20260926 03:58:36.608088 3779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39287 every 8 connection(s)
I20260926 03:58:36.608425 3651 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
I20260926 03:58:36.614007 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 3651
I20260926 03:58:36.614148 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-1/wal/instance
I20260926 03:58:36.614346 392 external_mini_cluster.cc:1428] Setting key 9608ef6b169a18a52cc23db0d7a8c8a2
Sep 26 03:58:36 dist-test-slave-nhpw krb5kdc[3410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395116, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 03:58:36.618451 3495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:36.609197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60789 (local address 127.0.98.62:35345)
0926 03:58:36.609243 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:36.609245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:36.609468 (+ 223us) server_negotiation.cc:407] Connection header received
0926 03:58:36.610002 (+ 534us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:36.610004 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:36.610016 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:36.610068 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:36.611028 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:36.611551 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:36.612584 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:36.612861 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:36.612950 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:36.615266 (+ 2316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:36.615268 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:36.615268 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:36.615277 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:36.616824 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:36.617350 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:36.617352 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:36.617352 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:36.617426 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:36.617736 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:36.617737 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:36.617738 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:36.617865 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:36.617942 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:36.618276 (+ 334us) server_negotiation.cc:299] Negotiation successful
0926 03:58:36.618361 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 03:58:36.618656 3782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:36.609175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35345 (local address 127.0.98.2:60789)
0926 03:58:36.609328 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:36.609370 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:36.609889 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:36.610148 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:36.610152 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:36.610727 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:36.610915 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:36.610920 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:36.611641 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:36.611642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:36.612471 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:36.612473 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:36.612959 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:36.612960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:36.613090 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:36.613607 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:36.613616 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:36.615169 (+ 1553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:58:36.616916 (+ 1747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:36.616923 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:36.616925 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:36.617251 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:36.617504 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:36.617505 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:36.617505 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:36.617645 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:36.617942 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:58:36.617947 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:58:36.618137 (+ 190us) client_negotiation.cc:769] Sending connection context
0926 03:58:36.618253 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 03:58:36.618409 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":74,"threads_started":1}
I20260926 03:58:36.619092 3780 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35345
I20260926 03:58:36.619202 3780 heartbeater.cc:464] Registering TS with master...
I20260926 03:58:36.619380 3780 heartbeater.cc:511] Master 127.0.98.62:35345 requested a full tablet report, sending...
I20260926 03:58:36.619823 3455 ts_manager.cc:194] Registered new tserver with Master: b39edf9305c043e4b14c77e63550430c (127.0.98.2:39287)
I20260926 03:58:36.620397 3455 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60789
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 03:58:36.645604 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35345
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:58:36.726387 3785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:58:36.726621 3785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:58:36.726657 3785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:58:36.727777 3785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:58:36.727845 3785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:58:36.727888 3785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 03:58:36.729391 3785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35345
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:58:36.729636 3785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:58:36.729856 3785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:58:36.734869 3790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:36.734931 3791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:36.735029 3793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:36.735037 3785 server_base.cc:1034] running on GCE node
I20260926 03:58:36.735357 3785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:58:36.735556 3785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:58:36.736701 3785 hybrid_clock.cc:648] HybridClock initialized: now 1790395116736689 us; error 26 us; skew 500 ppm
Sep 26 03:58:36 dist-test-slave-nhpw krb5kdc[3410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395116, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:58:36.739224 3785 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 03:58:36.739693 3785 webserver.cc:460] Webserver started at http://127.0.98.3:39791/ using document root <none> and password file <none>
I20260926 03:58:36.739951 3785 fs_manager.cc:359] Metadata directory not provided
I20260926 03:58:36.740031 3785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:58:36.740163 3785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:58:36.741254 3785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-2/data/instance:
uuid: "a02864ebe16344148259ae8b9d7e1b2d"
format_stamp: "Formatted at 2026-09-26 03:58:36 on dist-test-slave-nhpw"
server_key: "8b0265adb49c40ceaba148233ae9fbfc"
server_key_iv: "e2adaff354d4a7b93c9ac0d79c5ebbdc"
server_key_version: "encryptionkey@0"
I20260926 03:58:36.741631 3785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a02864ebe16344148259ae8b9d7e1b2d"
format_stamp: "Formatted at 2026-09-26 03:58:36 on dist-test-slave-nhpw"
server_key: "8b0265adb49c40ceaba148233ae9fbfc"
server_key_iv: "e2adaff354d4a7b93c9ac0d79c5ebbdc"
server_key_version: "encryptionkey@0"
I20260926 03:58:36.742967 3785 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 03:58:36.743762 3800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:36.743934 3785 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 03:58:36.744029 3785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-2/wal
uuid: "a02864ebe16344148259ae8b9d7e1b2d"
format_stamp: "Formatted at 2026-09-26 03:58:36 on dist-test-slave-nhpw"
server_key: "8b0265adb49c40ceaba148233ae9fbfc"
server_key_iv: "e2adaff354d4a7b93c9ac0d79c5ebbdc"
server_key_version: "encryptionkey@0"
I20260926 03:58:36.744112 3785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:58:36.761111 3785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:58:36.763075 3785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:58:36.763234 3785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:58:36.763456 3785 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:58:36.763757 3785 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:58:36.763810 3785 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:36.763849 3785 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:58:36.763877 3785 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:36.770030 3785 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40655
I20260926 03:58:36.770076 3913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40655 every 8 connection(s)
I20260926 03:58:36.770385 3785 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
Sep 26 03:58:36 dist-test-slave-nhpw krb5kdc[3410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395116, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 03:58:36.779181 3495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:36.771005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46291 (local address 127.0.98.62:35345)
0926 03:58:36.771048 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:36.771050 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:36.771251 (+ 201us) server_negotiation.cc:407] Connection header received
0926 03:58:36.771750 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:36.771752 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:36.771762 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:36.771817 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:36.772723 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:36.773266 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:36.774406 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:36.774588 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:36.774656 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:36.776815 (+ 2159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:36.776816 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:36.776816 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:36.776823 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:36.777993 (+ 1170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:36.778380 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:36.778380 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:36.778381 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:36.778429 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:36.778647 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:36.778647 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:36.778648 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:36.778744 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:36.778795 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:36.779048 (+ 253us) server_negotiation.cc:299] Negotiation successful
0926 03:58:36.779118 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 03:58:36.779410 3916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:36.771012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35345 (local address 127.0.98.3:46291)
0926 03:58:36.771157 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:36.771190 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:36.771655 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:36.771865 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:36.771868 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:36.772406 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:36.772594 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:36.772598 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:36.773342 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:36.773343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:36.774299 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:36.774302 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:36.774678 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:36.774678 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:36.774777 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:36.775213 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:36.775221 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:36.776731 (+ 1510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:58:36.778069 (+ 1338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:36.778075 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:36.778076 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:36.778312 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:36.778490 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:36.778491 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:36.778491 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:36.778590 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:36.778801 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:58:36.778804 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:58:36.778924 (+ 120us) client_negotiation.cc:769] Sending connection context
0926 03:58:36.779021 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 03:58:36.779147 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":91,"threads_started":1}
I20260926 03:58:36.779919 3914 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35345
I20260926 03:58:36.780045 3914 heartbeater.cc:464] Registering TS with master...
I20260926 03:58:36.780226 3914 heartbeater.cc:511] Master 127.0.98.62:35345 requested a full tablet report, sending...
I20260926 03:58:36.780416 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 3785
I20260926 03:58:36.780521 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790395095474643-392-0/minicluster-data/ts-2/wal/instance
I20260926 03:58:36.780646 3455 ts_manager.cc:194] Registered new tserver with Master: a02864ebe16344148259ae8b9d7e1b2d (127.0.98.3:40655)
I20260926 03:58:36.780720 392 external_mini_cluster.cc:1428] Setting key a1284f879eb66ae4818b620910c3d1d6
I20260926 03:58:36.781138 3455 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46291
I20260926 03:58:36.781591 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 03:58:36 dist-test-slave-nhpw krb5kdc[3410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395116, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 26 03:58:36 dist-test-slave-nhpw krb5kdc[3410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395116, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 03:58:36.803913 3495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:36.796821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48240 (local address 127.0.98.62:35345)
0926 03:58:36.796884 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:36.796886 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:36.796968 (+ 82us) server_negotiation.cc:407] Connection header received
0926 03:58:36.797006 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:36.797007 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:36.797015 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:36.797073 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:36.797609 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:36.798132 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:36.798826 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:36.799040 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:36.799097 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:36.801272 (+ 2175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:36.801274 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:36.801274 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:36.801280 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:36.802722 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:36.803108 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:36.803109 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:36.803109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:36.803168 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:36.803359 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:36.803360 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:36.803360 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:36.803508 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:36.803612 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:36.803731 (+ 119us) server_negotiation.cc:299] Negotiation successful
0926 03:58:36.803806 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
W20260926 03:58:36.804142 3455 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:48240
I20260926 03:58:36.804517 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 3517
I20260926 03:58:36.808521 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 3651
I20260926 03:58:36.812919 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 3785
I20260926 03:58:36.817176 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 3423
2026-09-26T03:58:36Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1846 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:58:36 dist-test-slave-nhpw krb5kdc[3926](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:58:36 dist-test-slave-nhpw krb5kdc[3926](info): set up 2 sockets
Sep 26 03:58:36 dist-test-slave-nhpw krb5kdc[3926](info): commencing operation
krb5kdc: starting...
W20260926 03:58:37.856801 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/krb5kdc/test-user.keytab.
Sep 26 03:58:37 dist-test-slave-nhpw krb5kdc[3926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395117, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:58:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:58:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 03:58:37.975975 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42011
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42011
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:58:38.055785 3939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:58:38.056015 3939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:58:38.056037 3939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:58:38.057271 3939 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:58:38.057317 3939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:58:38.057331 3939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:58:38.057343 3939 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:58:38.057356 3939 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:58:38.058857 3939 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42011
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:58:38.059109 3939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:58:38.059365 3939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:58:38.064098 3944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:38.064128 3945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:38.064339 3939 server_base.cc:1034] running on GCE node
W20260926 03:58:38.064109 3947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:38.064702 3939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:58:38.064949 3939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:58:38.066116 3939 hybrid_clock.cc:648] HybridClock initialized: now 1790395118066103 us; error 31 us; skew 500 ppm
Sep 26 03:58:38 dist-test-slave-nhpw krb5kdc[3926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395118, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:58:38.068553 3939 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 03:58:38.068994 3939 webserver.cc:460] Webserver started at http://127.0.98.62:40585/ using document root <none> and password file <none>
I20260926 03:58:38.069175 3939 fs_manager.cc:359] Metadata directory not provided
I20260926 03:58:38.069214 3939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:58:38.069298 3939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:58:38.070328 3939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/minicluster-data/master-0/data/instance:
uuid: "1a24fdae39eb486db4e0b3ebb8e98726"
format_stamp: "Formatted at 2026-09-26 03:58:38 on dist-test-slave-nhpw"
server_key: "0f64082596fab8ffef9009af7a88c92b"
server_key_iv: "108acef525df7aeafaa32e28b469312a"
server_key_version: "encryptionkey@0"
I20260926 03:58:38.070621 3939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/minicluster-data/master-0/wal/instance:
uuid: "1a24fdae39eb486db4e0b3ebb8e98726"
format_stamp: "Formatted at 2026-09-26 03:58:38 on dist-test-slave-nhpw"
server_key: "0f64082596fab8ffef9009af7a88c92b"
server_key_iv: "108acef525df7aeafaa32e28b469312a"
server_key_version: "encryptionkey@0"
I20260926 03:58:38.071820 3939 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 03:58:38.072448 3954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:38.072671 3939 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 03:58:38.072777 3939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/minicluster-data/master-0/wal
uuid: "1a24fdae39eb486db4e0b3ebb8e98726"
format_stamp: "Formatted at 2026-09-26 03:58:38 on dist-test-slave-nhpw"
server_key: "0f64082596fab8ffef9009af7a88c92b"
server_key_iv: "108acef525df7aeafaa32e28b469312a"
server_key_version: "encryptionkey@0"
I20260926 03:58:38.072856 3939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:58:38.082696 3939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:58:38.084496 3939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:58:38.084722 3939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:58:38.088986 3939 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42011
I20260926 03:58:38.089017 4006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42011 every 8 connection(s)
I20260926 03:58:38.089361 3939 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
I20260926 03:58:38.090224 4007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:58:38.090477 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 3939
I20260926 03:58:38.090584 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/minicluster-data/master-0/wal/instance
I20260926 03:58:38.090757 392 external_mini_cluster.cc:1428] Setting key 254e220fbcd092d5c5ba238550a2e301
I20260926 03:58:38.093743 4007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a24fdae39eb486db4e0b3ebb8e98726: Bootstrap starting.
I20260926 03:58:38.094465 4007 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a24fdae39eb486db4e0b3ebb8e98726: Neither blocks nor log segments found. Creating new log.
I20260926 03:58:38.094770 4007 log.cc:824] T 00000000000000000000000000000000 P 1a24fdae39eb486db4e0b3ebb8e98726: Log is configured to *not* fsync() on all Append() calls
I20260926 03:58:38.095577 4007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a24fdae39eb486db4e0b3ebb8e98726: No bootstrap required, opened a new log
Sep 26 03:58:38 dist-test-slave-nhpw krb5kdc[3926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395117, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 03:58:38.097723 4007 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a24fdae39eb486db4e0b3ebb8e98726 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a24fdae39eb486db4e0b3ebb8e98726" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42011 } }
I20260926 03:58:38.097864 4007 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a24fdae39eb486db4e0b3ebb8e98726 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:58:38.097895 4007 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a24fdae39eb486db4e0b3ebb8e98726 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a24fdae39eb486db4e0b3ebb8e98726, State: Initialized, Role: FOLLOWER
I20260926 03:58:38.098090 4007 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a24fdae39eb486db4e0b3ebb8e98726 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a24fdae39eb486db4e0b3ebb8e98726" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42011 } }
I20260926 03:58:38.098201 4007 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a24fdae39eb486db4e0b3ebb8e98726 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:58:38.098225 4007 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a24fdae39eb486db4e0b3ebb8e98726 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:58:38.098290 4007 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a24fdae39eb486db4e0b3ebb8e98726 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:58:38.098850 4007 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a24fdae39eb486db4e0b3ebb8e98726 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a24fdae39eb486db4e0b3ebb8e98726" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42011 } }
I20260926 03:58:38.098984 4007 leader_election.cc:302] T 00000000000000000000000000000000 P 1a24fdae39eb486db4e0b3ebb8e98726 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a24fdae39eb486db4e0b3ebb8e98726; no voters:
I20260926 03:58:38.099162 4007 leader_election.cc:288] T 00000000000000000000000000000000 P 1a24fdae39eb486db4e0b3ebb8e98726 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:58:38.099247 4012 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a24fdae39eb486db4e0b3ebb8e98726 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:58:38.099440 4007 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a24fdae39eb486db4e0b3ebb8e98726 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:58:38.099464 4012 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a24fdae39eb486db4e0b3ebb8e98726 [term 1 LEADER]: Becoming Leader. State: Replica: 1a24fdae39eb486db4e0b3ebb8e98726, State: Running, Role: LEADER
I20260926 03:58:38.099597 4012 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a24fdae39eb486db4e0b3ebb8e98726 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a24fdae39eb486db4e0b3ebb8e98726" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42011 } }
I20260926 03:58:38.099996 4014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a24fdae39eb486db4e0b3ebb8e98726 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a24fdae39eb486db4e0b3ebb8e98726. Latest consensus state: current_term: 1 leader_uuid: "1a24fdae39eb486db4e0b3ebb8e98726" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a24fdae39eb486db4e0b3ebb8e98726" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42011 } } }
I20260926 03:58:38.100229 4014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a24fdae39eb486db4e0b3ebb8e98726 [sys.catalog]: This master's current role is: LEADER
I20260926 03:58:38.100160 4013 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a24fdae39eb486db4e0b3ebb8e98726 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a24fdae39eb486db4e0b3ebb8e98726" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a24fdae39eb486db4e0b3ebb8e98726" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42011 } } }
I20260926 03:58:38.100489 4013 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a24fdae39eb486db4e0b3ebb8e98726 [sys.catalog]: This master's current role is: LEADER
I20260926 03:58:38.100658 4017 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:58:38.101186 4017 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:58:38.102608 4010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:38.091284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51410 (local address 127.0.98.62:42011)
0926 03:58:38.091459 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:38.091466 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:38.091491 (+ 25us) server_negotiation.cc:407] Connection header received
0926 03:58:38.092019 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:38.092026 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:38.092075 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:38.092204 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:38.093095 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:38.094071 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:38.095157 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:38.095680 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:38.095833 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:38.098652 (+ 2819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:38.098656 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:38.098656 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:38.098666 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:38.100862 (+ 2196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:38.101283 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:38.101286 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:38.101287 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:38.101394 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:38.101676 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:38.101677 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:38.101677 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:38.101987 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:38.102148 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:38.102267 (+ 119us) server_negotiation.cc:299] Negotiation successful
0926 03:58:38.102401 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":77,"threads_started":1}
I20260926 03:58:38.102944 4017 catalog_manager.cc:1347] Generated new cluster ID: 780787d2a4b4495d8efc9976f658e4b4
I20260926 03:58:38.102990 4017 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:58:38.119437 4017 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:58:38.119966 4017 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:58:38.126041 4017 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a24fdae39eb486db4e0b3ebb8e98726: Generated new TSK 0
I20260926 03:58:38.126245 4017 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 03:58:38.151826 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42011
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:58:38.233870 4033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:58:38.234117 4033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:58:38.234156 4033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:58:38.235285 4033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:58:38.235356 4033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:58:38.235399 4033 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 03:58:38.237597 4033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:58:38.237880 4033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:58:38.238116 4033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:58:38.243103 4039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:38.243121 4038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:38.243093 4041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:38.243322 4033 server_base.cc:1034] running on GCE node
I20260926 03:58:38.243618 4033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:58:38.243861 4033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:58:38.245033 4033 hybrid_clock.cc:648] HybridClock initialized: now 1790395118245007 us; error 38 us; skew 500 ppm
Sep 26 03:58:38 dist-test-slave-nhpw krb5kdc[3926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395118, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:58:38.247666 4033 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 03:58:38.248129 4033 webserver.cc:460] Webserver started at http://127.0.98.1:43849/ using document root <none> and password file <none>
I20260926 03:58:38.248364 4033 fs_manager.cc:359] Metadata directory not provided
I20260926 03:58:38.248409 4033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:58:38.248556 4033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:58:38.249604 4033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/minicluster-data/ts-0/data/instance:
uuid: "867733e4ab2a48849a47eb08bf4e09a1"
format_stamp: "Formatted at 2026-09-26 03:58:38 on dist-test-slave-nhpw"
server_key: "71151a7c35ea04ee5229169db8e9b9ca"
server_key_iv: "871ae1b86e0fe246e3175c36ab497945"
server_key_version: "encryptionkey@0"
I20260926 03:58:38.249959 4033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/minicluster-data/ts-0/wal/instance:
uuid: "867733e4ab2a48849a47eb08bf4e09a1"
format_stamp: "Formatted at 2026-09-26 03:58:38 on dist-test-slave-nhpw"
server_key: "71151a7c35ea04ee5229169db8e9b9ca"
server_key_iv: "871ae1b86e0fe246e3175c36ab497945"
server_key_version: "encryptionkey@0"
I20260926 03:58:38.251206 4033 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 03:58:38.251906 4048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:38.252110 4033 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 03:58:38.252190 4033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/minicluster-data/ts-0/wal
uuid: "867733e4ab2a48849a47eb08bf4e09a1"
format_stamp: "Formatted at 2026-09-26 03:58:38 on dist-test-slave-nhpw"
server_key: "71151a7c35ea04ee5229169db8e9b9ca"
server_key_iv: "871ae1b86e0fe246e3175c36ab497945"
server_key_version: "encryptionkey@0"
I20260926 03:58:38.252235 4033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:58:38.272123 4033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:58:38.274365 4033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:58:38.274478 4033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:58:38.274676 4033 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:58:38.274995 4033 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:58:38.275031 4033 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:38.275084 4033 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:58:38.275110 4033 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:38.281069 4033 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33051
I20260926 03:58:38.281162 4161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33051 every 8 connection(s)
I20260926 03:58:38.281487 4033 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
I20260926 03:58:38.287009 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 4033
I20260926 03:58:38.287139 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/minicluster-data/ts-0/wal/instance
I20260926 03:58:38.287345 392 external_mini_cluster.cc:1428] Setting key 5b3f30561fc02ec478033cb792c393e0
Sep 26 03:58:38 dist-test-slave-nhpw krb5kdc[3926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395118, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 03:58:38.291473 4010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:38.282276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60309 (local address 127.0.98.62:42011)
0926 03:58:38.282357 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:38.282359 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:38.282543 (+ 184us) server_negotiation.cc:407] Connection header received
0926 03:58:38.283470 (+ 927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:38.283471 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:38.283481 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:38.283529 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:38.284126 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:38.284623 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:38.285656 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:38.285847 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:38.285897 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:38.288354 (+ 2457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:38.288355 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:38.288356 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:38.288363 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:38.289974 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:38.290501 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:38.290503 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:38.290503 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:38.290560 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:38.290793 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:38.290794 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:38.290795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:38.290942 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:38.291011 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:38.291250 (+ 239us) server_negotiation.cc:299] Negotiation successful
0926 03:58:38.291340 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 03:58:38.291499 4164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:38.282235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42011 (local address 127.0.98.1:60309)
0926 03:58:38.282417 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:38.282453 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:38.283377 (+ 924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:38.283591 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:38.283595 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:38.283843 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:38.284043 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:38.284048 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:38.284744 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:38.284745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:38.285582 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:38.285584 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:38.285933 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:38.285933 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:38.286041 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:38.286624 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:38.286632 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:38.288262 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:58:38.290068 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:38.290075 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:38.290077 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:38.290414 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:38.290630 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:38.290630 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:38.290631 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:38.290730 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:38.291005 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:58:38.291009 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:58:38.291125 (+ 116us) client_negotiation.cc:769] Sending connection context
0926 03:58:38.291207 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 03:58:38.291327 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":92,"threads_started":1}
I20260926 03:58:38.292027 4162 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42011
I20260926 03:58:38.292155 4162 heartbeater.cc:464] Registering TS with master...
I20260926 03:58:38.292340 4162 heartbeater.cc:511] Master 127.0.98.62:42011 requested a full tablet report, sending...
I20260926 03:58:38.292917 3971 ts_manager.cc:194] Registered new tserver with Master: 867733e4ab2a48849a47eb08bf4e09a1 (127.0.98.1:33051)
I20260926 03:58:38.293540 3971 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60309
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 03:58:38.315318 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42011
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:58:38.404023 4167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:58:38.404291 4167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:58:38.404331 4167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:58:38.405572 4167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:58:38.405638 4167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:58:38.405682 4167 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 03:58:38.407155 4167 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4167
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:58:38.407434 4167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:58:38.407673 4167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:58:38.412750 4175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:38.412744 4173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:38.412753 4172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:38.412863 4167 server_base.cc:1034] running on GCE node
I20260926 03:58:38.413273 4167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:58:38.413537 4167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:58:38.414692 4167 hybrid_clock.cc:648] HybridClock initialized: now 1790395118414688 us; error 35 us; skew 500 ppm
Sep 26 03:58:38 dist-test-slave-nhpw krb5kdc[3926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395118, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:58:38.417297 4167 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 03:58:38.417706 4167 webserver.cc:460] Webserver started at http://127.0.98.2:37713/ using document root <none> and password file <none>
I20260926 03:58:38.417933 4167 fs_manager.cc:359] Metadata directory not provided
I20260926 03:58:38.418004 4167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:58:38.418166 4167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:58:38.419158 4167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/minicluster-data/ts-1/data/instance:
uuid: "3b51b9c03f924ea595bcd76a03733c03"
format_stamp: "Formatted at 2026-09-26 03:58:38 on dist-test-slave-nhpw"
server_key: "e5f9119bb43e0bf390ac273c5c8a8743"
server_key_iv: "e97a3fb85f93cd302a92d91c0e760b98"
server_key_version: "encryptionkey@0"
I20260926 03:58:38.419492 4167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3b51b9c03f924ea595bcd76a03733c03"
format_stamp: "Formatted at 2026-09-26 03:58:38 on dist-test-slave-nhpw"
server_key: "e5f9119bb43e0bf390ac273c5c8a8743"
server_key_iv: "e97a3fb85f93cd302a92d91c0e760b98"
server_key_version: "encryptionkey@0"
I20260926 03:58:38.420780 4167 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 03:58:38.421562 4182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:38.421761 4167 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 03:58:38.421856 4167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/minicluster-data/ts-1/wal
uuid: "3b51b9c03f924ea595bcd76a03733c03"
format_stamp: "Formatted at 2026-09-26 03:58:38 on dist-test-slave-nhpw"
server_key: "e5f9119bb43e0bf390ac273c5c8a8743"
server_key_iv: "e97a3fb85f93cd302a92d91c0e760b98"
server_key_version: "encryptionkey@0"
I20260926 03:58:38.421940 4167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:58:38.445662 4167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:58:38.448028 4167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:58:38.448190 4167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:58:38.448419 4167 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:58:38.448777 4167 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:58:38.448832 4167 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:38.448870 4167 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:58:38.448899 4167 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:38.454756 4167 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46457
I20260926 03:58:38.454777 4295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46457 every 8 connection(s)
I20260926 03:58:38.455133 4167 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
Sep 26 03:58:38 dist-test-slave-nhpw krb5kdc[3926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395118, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 03:58:38.461982 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 4167
I20260926 03:58:38.462105 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/minicluster-data/ts-1/wal/instance
I20260926 03:58:38.462291 392 external_mini_cluster.cc:1428] Setting key cfd33bb19e1421d9ba860d1676a0ad69
I20260926 03:58:38.464166 4010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:38.455802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43297 (local address 127.0.98.62:42011)
0926 03:58:38.455889 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:38.455892 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:38.456080 (+ 188us) server_negotiation.cc:407] Connection header received
0926 03:58:38.457003 (+ 923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:38.457004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:38.457014 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:38.457072 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:38.457634 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:38.458139 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:38.459089 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:38.459262 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:38.459313 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:38.461282 (+ 1969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:38.461284 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:38.461284 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:38.461290 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:38.462855 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:38.463246 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:38.463247 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:38.463248 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:38.463307 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:38.463554 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:38.463555 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:38.463556 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:38.463664 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:38.463739 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:38.463993 (+ 254us) server_negotiation.cc:299] Negotiation successful
0926 03:58:38.464078 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 03:58:38.464386 4298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:38.455766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42011 (local address 127.0.98.2:43297)
0926 03:58:38.455954 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:38.455996 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:38.456908 (+ 912us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:38.457137 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:38.457141 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:38.457370 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:38.457553 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:38.457557 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:38.458221 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:38.458222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:38.459016 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:38.459017 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:38.459331 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:38.459331 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:38.459424 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:38.459855 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:38.459862 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:38.461198 (+ 1336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:58:38.462926 (+ 1728us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:38.462933 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:38.462935 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:38.463150 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:38.463374 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:38.463375 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:38.463376 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:38.463476 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:38.463731 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:58:38.463736 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:58:38.463867 (+ 131us) client_negotiation.cc:769] Sending connection context
0926 03:58:38.463975 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 03:58:38.464123 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":120,"threads_started":1}
I20260926 03:58:38.464836 4296 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42011
I20260926 03:58:38.464926 4296 heartbeater.cc:464] Registering TS with master...
I20260926 03:58:38.465118 4296 heartbeater.cc:511] Master 127.0.98.62:42011 requested a full tablet report, sending...
I20260926 03:58:38.465488 3971 ts_manager.cc:194] Registered new tserver with Master: 3b51b9c03f924ea595bcd76a03733c03 (127.0.98.2:46457)
I20260926 03:58:38.465996 3971 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43297
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 03:58:38.487591 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42011
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36227
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:58:38.567878 4301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:58:38.568085 4301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:58:38.568109 4301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:58:38.569222 4301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:58:38.569260 4301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:58:38.569286 4301 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 03:58:38.570660 4301 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36227
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42011
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:58:38.570839 4301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:58:38.571019 4301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:58:38.576256 4309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:38.576309 4307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:38.576274 4306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:38.576484 4301 server_base.cc:1034] running on GCE node
I20260926 03:58:38.576714 4301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:58:38.576926 4301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:58:38.578100 4301 hybrid_clock.cc:648] HybridClock initialized: now 1790395118578091 us; error 39 us; skew 500 ppm
Sep 26 03:58:38 dist-test-slave-nhpw krb5kdc[3926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395118, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:58:38.580753 4301 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 03:58:38.581260 4301 webserver.cc:460] Webserver started at http://127.0.98.3:43353/ using document root <none> and password file <none>
I20260926 03:58:38.581504 4301 fs_manager.cc:359] Metadata directory not provided
I20260926 03:58:38.581574 4301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:58:38.581727 4301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:58:38.582762 4301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/minicluster-data/ts-2/data/instance:
uuid: "f8f99ab31f0a4e4fb8320e5abd2d856c"
format_stamp: "Formatted at 2026-09-26 03:58:38 on dist-test-slave-nhpw"
server_key: "899bc78cf36bbf0a59797f8a393ddcfd"
server_key_iv: "3a8d2842440cfb16be8ecc8de8bf13a5"
server_key_version: "encryptionkey@0"
I20260926 03:58:38.583125 4301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f8f99ab31f0a4e4fb8320e5abd2d856c"
format_stamp: "Formatted at 2026-09-26 03:58:38 on dist-test-slave-nhpw"
server_key: "899bc78cf36bbf0a59797f8a393ddcfd"
server_key_iv: "3a8d2842440cfb16be8ecc8de8bf13a5"
server_key_version: "encryptionkey@0"
I20260926 03:58:38.584451 4301 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 03:58:38.585268 4316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:38.585433 4301 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 03:58:38.585529 4301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/minicluster-data/ts-2/wal
uuid: "f8f99ab31f0a4e4fb8320e5abd2d856c"
format_stamp: "Formatted at 2026-09-26 03:58:38 on dist-test-slave-nhpw"
server_key: "899bc78cf36bbf0a59797f8a393ddcfd"
server_key_iv: "3a8d2842440cfb16be8ecc8de8bf13a5"
server_key_version: "encryptionkey@0"
I20260926 03:58:38.585613 4301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:58:38.597159 4301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:58:38.599557 4301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:58:38.599725 4301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:58:38.599928 4301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:58:38.600219 4301 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:58:38.600250 4301 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:38.600272 4301 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:58:38.600286 4301 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:38.606309 4301 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43467
I20260926 03:58:38.606350 4429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43467 every 8 connection(s)
I20260926 03:58:38.606669 4301 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
I20260926 03:58:38.612002 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 4301
I20260926 03:58:38.612085 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790395095474643-392-0/minicluster-data/ts-2/wal/instance
I20260926 03:58:38.612243 392 external_mini_cluster.cc:1428] Setting key a3b1eda6d9419520735355a01317f6d7
Sep 26 03:58:38 dist-test-slave-nhpw krb5kdc[3926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395118, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 03:58:38.615921 4010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:38.607397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55809 (local address 127.0.98.62:42011)
0926 03:58:38.607473 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:38.607475 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:38.607626 (+ 151us) server_negotiation.cc:407] Connection header received
0926 03:58:38.608509 (+ 883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:38.608510 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:38.608520 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:38.608585 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:38.609235 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:38.609732 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:38.610704 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:38.610882 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:38.610930 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:38.613313 (+ 2383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:38.613314 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:38.613315 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:38.613321 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:38.614763 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:38.615098 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:38.615099 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:38.615099 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:38.615147 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:38.615377 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:38.615379 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:38.615379 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:38.615498 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:38.615566 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:38.615768 (+ 202us) server_negotiation.cc:299] Negotiation successful
0926 03:58:38.615829 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 03:58:38.616089 4432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:38.607349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42011 (local address 127.0.98.3:55809)
0926 03:58:38.607500 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:38.607535 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:38.608422 (+ 887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:38.608699 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:38.608703 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:38.608954 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:38.609160 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:38.609165 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:38.609811 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:38.609812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:38.610633 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:38.610635 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:38.610954 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:38.610955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:38.611049 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:38.611576 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:38.611583 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:38.613238 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:58:38.614830 (+ 1592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:38.614834 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:38.614836 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:38.615035 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:38.615200 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:38.615201 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:38.615201 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:38.615289 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:38.615571 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:58:38.615574 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:58:38.615677 (+ 103us) client_negotiation.cc:769] Sending connection context
0926 03:58:38.615753 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 03:58:38.615875 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":91,"threads_started":1}
I20260926 03:58:38.616532 4430 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42011
I20260926 03:58:38.616631 4430 heartbeater.cc:464] Registering TS with master...
I20260926 03:58:38.616817 4430 heartbeater.cc:511] Master 127.0.98.62:42011 requested a full tablet report, sending...
I20260926 03:58:38.617205 3971 ts_manager.cc:194] Registered new tserver with Master: f8f99ab31f0a4e4fb8320e5abd2d856c (127.0.98.3:43467)
I20260926 03:58:38.617574 3971 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55809
I20260926 03:58:38.624018 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 03:58:38 dist-test-slave-nhpw krb5kdc[3926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 03:58:38 dist-test-slave-nhpw krb5kdc[3926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 03:58:38.648478 4437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:38.640529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35494 (local address 127.0.98.1:33051)
0926 03:58:38.640752 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:38.640757 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:38.640767 (+ 10us) server_negotiation.cc:407] Connection header received
0926 03:58:38.640795 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:38.640797 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:38.640811 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:38.640878 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:38.641451 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:38.642206 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:38.643021 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:38.643363 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:38.643446 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:38.645736 (+ 2290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:38.645755 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:38.645755 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:38.645762 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:38.647408 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:38.647741 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:38.647744 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:38.647745 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:38.647804 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:38.647972 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:38.647973 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:38.647973 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:38.648100 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:38.648210 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:38.648267 (+ 57us) server_negotiation.cc:299] Negotiation successful
0926 03:58:38.648331 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":132,"threads_started":1}
Sep 26 03:58:38 dist-test-slave-nhpw krb5kdc[3926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395118, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 03:58:38.656556 4010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:38.649851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51414 (local address 127.0.98.62:42011)
0926 03:58:38.649913 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:38.649915 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:38.649946 (+ 31us) server_negotiation.cc:407] Connection header received
0926 03:58:38.649969 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:38.649970 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:38.649979 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:38.650026 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:38.650433 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:38.650899 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:38.651588 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:38.651774 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:38.651839 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:38.653992 (+ 2153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:38.653994 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:38.653994 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:38.654001 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:38.655471 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:38.655816 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:38.655817 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:38.655818 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:38.655883 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:38.656057 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:38.656057 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:38.656057 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:38.656186 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:38.656257 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:38.656412 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 03:58:38.656469 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
W20260926 03:58:38.656863 3971 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51414
I20260926 03:58:38.657269 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 4033
I20260926 03:58:38.662809 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 4167
I20260926 03:58:38.667002 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 4301
I20260926 03:58:38.671043 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 3939
2026-09-26T03:58:38Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1853 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:58:38 dist-test-slave-nhpw krb5kdc[4442](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:58:38 dist-test-slave-nhpw krb5kdc[4442](info): set up 2 sockets
Sep 26 03:58:38 dist-test-slave-nhpw krb5kdc[4442](info): commencing operation
krb5kdc: starting...
W20260926 03:58:39.700464 392 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/krb5kdc/test-user.keytab.
Sep 26 03:58:39 dist-test-slave-nhpw krb5kdc[4442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:58:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:58:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 03:58:39.816861 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45331
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45331
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:58:39.895231 4455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:58:39.895419 4455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:58:39.895442 4455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:58:39.896734 4455 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:58:39.896776 4455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:58:39.896790 4455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:58:39.896802 4455 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:58:39.896852 4455 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:58:39.898296 4455 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45331
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:58:39.898545 4455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:58:39.898768 4455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:58:39.903715 4461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:39.903752 4460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:39.903807 4455 server_base.cc:1034] running on GCE node
W20260926 03:58:39.903718 4463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:39.904176 4455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:58:39.904474 4455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:58:39.905669 4455 hybrid_clock.cc:648] HybridClock initialized: now 1790395119905660 us; error 54 us; skew 500 ppm
Sep 26 03:58:39 dist-test-slave-nhpw krb5kdc[4442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395119, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:58:39.913736 4455 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 03:58:39.914176 4455 fs_manager.cc:359] Metadata directory not provided
I20260926 03:58:39.914252 4455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:58:39.914410 4455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:58:39.915506 4455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/minicluster-data/master-0/data/instance:
uuid: "17e0a77798b844359665d39d3e99903c"
format_stamp: "Formatted at 2026-09-26 03:58:39 on dist-test-slave-nhpw"
server_key: "244a248ad27ac24d14ffc2ee4470f9b5"
server_key_iv: "ecfb155a94aabc61eb93a8f92373b2a4"
server_key_version: "encryptionkey@0"
I20260926 03:58:39.915879 4455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/minicluster-data/master-0/wal/instance:
uuid: "17e0a77798b844359665d39d3e99903c"
format_stamp: "Formatted at 2026-09-26 03:58:39 on dist-test-slave-nhpw"
server_key: "244a248ad27ac24d14ffc2ee4470f9b5"
server_key_iv: "ecfb155a94aabc61eb93a8f92373b2a4"
server_key_version: "encryptionkey@0"
I20260926 03:58:39.917323 4455 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 03:58:39.918150 4469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:39.918335 4455 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 03:58:39.918443 4455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/minicluster-data/master-0/wal
uuid: "17e0a77798b844359665d39d3e99903c"
format_stamp: "Formatted at 2026-09-26 03:58:39 on dist-test-slave-nhpw"
server_key: "244a248ad27ac24d14ffc2ee4470f9b5"
server_key_iv: "ecfb155a94aabc61eb93a8f92373b2a4"
server_key_version: "encryptionkey@0"
I20260926 03:58:39.918519 4455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:58:39.935328 4455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:58:39.937654 4455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:58:39.937817 4455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:58:39.941679 4455 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45331
I20260926 03:58:39.941706 4521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45331 every 8 connection(s)
I20260926 03:58:39.941979 4455 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
I20260926 03:58:39.943387 4522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:58:39.946236 4522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17e0a77798b844359665d39d3e99903c: Bootstrap starting.
I20260926 03:58:39.946813 4522 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17e0a77798b844359665d39d3e99903c: Neither blocks nor log segments found. Creating new log.
I20260926 03:58:39.947113 4522 log.cc:824] T 00000000000000000000000000000000 P 17e0a77798b844359665d39d3e99903c: Log is configured to *not* fsync() on all Append() calls
I20260926 03:58:39.947755 4522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17e0a77798b844359665d39d3e99903c: No bootstrap required, opened a new log
I20260926 03:58:39.949388 4522 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17e0a77798b844359665d39d3e99903c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17e0a77798b844359665d39d3e99903c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45331 } }
I20260926 03:58:39.949520 4522 raft_consensus.cc:374] T 00000000000000000000000000000000 P 17e0a77798b844359665d39d3e99903c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:58:39.949542 4522 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17e0a77798b844359665d39d3e99903c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17e0a77798b844359665d39d3e99903c, State: Initialized, Role: FOLLOWER
I20260926 03:58:39.949709 4522 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17e0a77798b844359665d39d3e99903c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17e0a77798b844359665d39d3e99903c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45331 } }
I20260926 03:58:39.949769 4522 raft_consensus.cc:388] T 00000000000000000000000000000000 P 17e0a77798b844359665d39d3e99903c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:58:39.949808 4522 raft_consensus.cc:482] T 00000000000000000000000000000000 P 17e0a77798b844359665d39d3e99903c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:58:39.949855 4522 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17e0a77798b844359665d39d3e99903c [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:58:39.950659 4522 raft_consensus.cc:504] T 00000000000000000000000000000000 P 17e0a77798b844359665d39d3e99903c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17e0a77798b844359665d39d3e99903c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45331 } }
I20260926 03:58:39.950767 4522 leader_election.cc:302] T 00000000000000000000000000000000 P 17e0a77798b844359665d39d3e99903c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 17e0a77798b844359665d39d3e99903c; no voters:
I20260926 03:58:39.950937 4522 leader_election.cc:288] T 00000000000000000000000000000000 P 17e0a77798b844359665d39d3e99903c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:58:39.950974 4525 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 17e0a77798b844359665d39d3e99903c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:58:39.951160 4525 raft_consensus.cc:686] T 00000000000000000000000000000000 P 17e0a77798b844359665d39d3e99903c [term 1 LEADER]: Becoming Leader. State: Replica: 17e0a77798b844359665d39d3e99903c, State: Running, Role: LEADER
I20260926 03:58:39.951195 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 4455
I20260926 03:58:39.951236 4522 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17e0a77798b844359665d39d3e99903c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:58:39.951290 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/minicluster-data/master-0/wal/instance
I20260926 03:58:39.951239 4525 consensus_queue.cc:237] T 00000000000000000000000000000000 P 17e0a77798b844359665d39d3e99903c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17e0a77798b844359665d39d3e99903c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45331 } }
I20260926 03:58:39.951489 392 external_mini_cluster.cc:1428] Setting key 0e600ea0f850e8673ed5e8c46e5ad39f
I20260926 03:58:39.951618 4525 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17e0a77798b844359665d39d3e99903c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 17e0a77798b844359665d39d3e99903c. Latest consensus state: current_term: 1 leader_uuid: "17e0a77798b844359665d39d3e99903c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17e0a77798b844359665d39d3e99903c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45331 } } }
I20260926 03:58:39.951628 4527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17e0a77798b844359665d39d3e99903c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "17e0a77798b844359665d39d3e99903c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "17e0a77798b844359665d39d3e99903c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45331 } } }
I20260926 03:58:39.951694 4525 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17e0a77798b844359665d39d3e99903c [sys.catalog]: This master's current role is: LEADER
I20260926 03:58:39.951695 4527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17e0a77798b844359665d39d3e99903c [sys.catalog]: This master's current role is: LEADER
I20260926 03:58:39.951941 4531 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:58:39.952769 4531 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:58:39.954303 4531 catalog_manager.cc:1347] Generated new cluster ID: c741b4395e064e679c31719ce081c77c
I20260926 03:58:39.954355 4531 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 03:58:39 dist-test-slave-nhpw krb5kdc[4442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 03:58:39.961526 4536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:39.952098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40702 (local address 127.0.98.62:45331)
0926 03:58:39.952257 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:39.952267 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:39.952432 (+ 165us) server_negotiation.cc:407] Connection header received
0926 03:58:39.952658 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:39.952667 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:39.952725 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:39.952879 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:39.953551 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:39.954780 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:39.955563 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:39.956000 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:39.956138 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:39.958394 (+ 2256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:39.958398 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:39.958398 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:39.958409 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:39.960032 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:39.960403 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:39.960407 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:39.960407 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:39.960490 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:39.960722 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:39.960723 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:39.960723 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:39.960938 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:39.961113 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:39.961178 (+ 65us) server_negotiation.cc:299] Negotiation successful
0926 03:58:39.961313 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":69,"threads_started":1}
I20260926 03:58:39.975275 4531 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:58:39.975788 4531 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:58:39.977880 4531 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 17e0a77798b844359665d39d3e99903c: Generated new TSK 0
I20260926 03:58:39.978052 4531 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 03:58:40.007462 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45331
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:58:40.123340 4548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:58:40.123620 4548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:58:40.123656 4548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:58:40.125473 4548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:58:40.125533 4548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:58:40.125581 4548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 03:58:40.127833 4548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:58:40.128059 4548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:58:40.128291 4548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:58:40.133298 4554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:40.133395 4553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:40.133308 4556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:40.133601 4548 server_base.cc:1034] running on GCE node
I20260926 03:58:40.133775 4548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:58:40.134020 4548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:58:40.135215 4548 hybrid_clock.cc:648] HybridClock initialized: now 1790395120135201 us; error 27 us; skew 500 ppm
Sep 26 03:58:40 dist-test-slave-nhpw krb5kdc[4442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395120, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:58:40.138105 4548 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 03:58:40.138446 4548 fs_manager.cc:359] Metadata directory not provided
I20260926 03:58:40.138495 4548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:58:40.138623 4548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:58:40.139662 4548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/minicluster-data/ts-0/data/instance:
uuid: "c4e698cb5e63464c926f0c3c6ecf508c"
format_stamp: "Formatted at 2026-09-26 03:58:40 on dist-test-slave-nhpw"
server_key: "edf3768adc447e136b084fb5c0fe95cb"
server_key_iv: "422c0c5040c8f7a333d8cdd56a474b34"
server_key_version: "encryptionkey@0"
I20260926 03:58:40.140021 4548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c4e698cb5e63464c926f0c3c6ecf508c"
format_stamp: "Formatted at 2026-09-26 03:58:40 on dist-test-slave-nhpw"
server_key: "edf3768adc447e136b084fb5c0fe95cb"
server_key_iv: "422c0c5040c8f7a333d8cdd56a474b34"
server_key_version: "encryptionkey@0"
I20260926 03:58:40.141422 4548 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 03:58:40.142129 4562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:40.142323 4548 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 03:58:40.142397 4548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/minicluster-data/ts-0/wal
uuid: "c4e698cb5e63464c926f0c3c6ecf508c"
format_stamp: "Formatted at 2026-09-26 03:58:40 on dist-test-slave-nhpw"
server_key: "edf3768adc447e136b084fb5c0fe95cb"
server_key_iv: "422c0c5040c8f7a333d8cdd56a474b34"
server_key_version: "encryptionkey@0"
I20260926 03:58:40.142443 4548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:58:40.152236 4548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:58:40.154172 4548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:58:40.154320 4548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:58:40.154564 4548 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:58:40.154873 4548 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:58:40.154929 4548 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:40.154970 4548 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:58:40.155001 4548 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:40.160406 4548 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37905
I20260926 03:58:40.160471 4675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37905 every 8 connection(s)
I20260926 03:58:40.160804 4548 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
I20260926 03:58:40.162990 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 4548
I20260926 03:58:40.163120 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/minicluster-data/ts-0/wal/instance
I20260926 03:58:40.163302 392 external_mini_cluster.cc:1428] Setting key c7d95ca0f66e54394122659fead4bfe1
Sep 26 03:58:40 dist-test-slave-nhpw krb5kdc[4442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395120, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 03:58:40.171121 4536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:40.161501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37971 (local address 127.0.98.62:45331)
0926 03:58:40.161548 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:40.161551 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:40.161718 (+ 167us) server_negotiation.cc:407] Connection header received
0926 03:58:40.162243 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:40.162245 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:40.162262 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:40.162326 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:40.163470 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:40.164098 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:40.165587 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:40.165803 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:40.165889 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:40.168276 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:40.168277 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:40.168278 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:40.168286 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:40.169705 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:40.170201 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:40.170201 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:40.170202 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:40.170248 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:40.170515 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:40.170515 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:40.170515 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:40.170615 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:40.170663 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:40.170988 (+ 325us) server_negotiation.cc:299] Negotiation successful
0926 03:58:40.171055 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 03:58:40.171423 4678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:40.161461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45331 (local address 127.0.98.1:37971)
0926 03:58:40.161593 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:40.161636 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:40.162139 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:40.162393 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:40.162397 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:40.162651 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:40.163326 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:40.163334 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:40.164185 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:40.164186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:40.165479 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:40.165482 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:40.165897 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:40.165898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:40.166027 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:40.166506 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:40.166514 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:40.168179 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:58:40.169787 (+ 1608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:40.169793 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:40.169795 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:40.170116 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:40.170311 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:40.170312 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:40.170312 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:40.170446 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:40.170683 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:58:40.170689 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:58:40.170846 (+ 157us) client_negotiation.cc:769] Sending connection context
0926 03:58:40.170973 (+ 127us) client_negotiation.cc:240] Negotiation successful
0926 03:58:40.171153 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":62,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 03:58:40.171967 4676 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45331
I20260926 03:58:40.172058 4676 heartbeater.cc:464] Registering TS with master...
I20260926 03:58:40.172235 4676 heartbeater.cc:511] Master 127.0.98.62:45331 requested a full tablet report, sending...
I20260926 03:58:40.172875 4486 ts_manager.cc:194] Registered new tserver with Master: c4e698cb5e63464c926f0c3c6ecf508c (127.0.98.1:37905)
I20260926 03:58:40.173450 4486 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37971
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 03:58:40.187937 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45331
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:58:40.267645 4681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:58:40.267846 4681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:58:40.267869 4681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:58:40.268972 4681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:58:40.269007 4681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:58:40.269032 4681 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 03:58:40.270423 4681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:58:40.270606 4681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:58:40.270781 4681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:58:40.275579 4687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:40.275641 4686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:40.275727 4681 server_base.cc:1034] running on GCE node
W20260926 03:58:40.275631 4689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:40.275995 4681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:58:40.276239 4681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:58:40.277397 4681 hybrid_clock.cc:648] HybridClock initialized: now 1790395120277395 us; error 36 us; skew 500 ppm
Sep 26 03:58:40 dist-test-slave-nhpw krb5kdc[4442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395120, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:58:40.279867 4681 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 03:58:40.280208 4681 fs_manager.cc:359] Metadata directory not provided
I20260926 03:58:40.280256 4681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:58:40.280346 4681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:58:40.281375 4681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/minicluster-data/ts-1/data/instance:
uuid: "27fb70bd8119450583135702b45e5a38"
format_stamp: "Formatted at 2026-09-26 03:58:40 on dist-test-slave-nhpw"
server_key: "e2776483e8c2dacbb9bc61b73e6a3599"
server_key_iv: "7c29569780ec78d99ef3e7b353325645"
server_key_version: "encryptionkey@0"
I20260926 03:58:40.281682 4681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/minicluster-data/ts-1/wal/instance:
uuid: "27fb70bd8119450583135702b45e5a38"
format_stamp: "Formatted at 2026-09-26 03:58:40 on dist-test-slave-nhpw"
server_key: "e2776483e8c2dacbb9bc61b73e6a3599"
server_key_iv: "7c29569780ec78d99ef3e7b353325645"
server_key_version: "encryptionkey@0"
I20260926 03:58:40.282894 4681 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 03:58:40.283569 4695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:40.283773 4681 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 03:58:40.283859 4681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/minicluster-data/ts-1/wal
uuid: "27fb70bd8119450583135702b45e5a38"
format_stamp: "Formatted at 2026-09-26 03:58:40 on dist-test-slave-nhpw"
server_key: "e2776483e8c2dacbb9bc61b73e6a3599"
server_key_iv: "7c29569780ec78d99ef3e7b353325645"
server_key_version: "encryptionkey@0"
I20260926 03:58:40.283907 4681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:58:40.311983 4681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:58:40.313956 4681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:58:40.314090 4681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:58:40.314281 4681 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:58:40.314622 4681 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:58:40.314659 4681 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:40.314682 4681 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:58:40.314743 4681 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:40.320389 4681 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38299
I20260926 03:58:40.320443 4808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38299 every 8 connection(s)
I20260926 03:58:40.320811 4681 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
I20260926 03:58:40.322732 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 4681
I20260926 03:58:40.322816 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/minicluster-data/ts-1/wal/instance
I20260926 03:58:40.323047 392 external_mini_cluster.cc:1428] Setting key c85d4ea9c2e8f0e193964b9d14401fb3
Sep 26 03:58:40 dist-test-slave-nhpw krb5kdc[4442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395120, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 03:58:40.331271 4536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:40.321569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44989 (local address 127.0.98.62:45331)
0926 03:58:40.321639 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:40.321642 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:40.321869 (+ 227us) server_negotiation.cc:407] Connection header received
0926 03:58:40.322446 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:40.322447 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:40.322457 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:40.322519 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:40.323663 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:40.324258 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:40.325639 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:40.325972 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:40.326059 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:40.328431 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:40.328433 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:40.328433 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:40.328440 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:40.329896 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:40.330314 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:40.330316 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:40.330316 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:40.330374 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:40.330639 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:40.330640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:40.330640 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:40.330751 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:40.330828 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:40.331095 (+ 267us) server_negotiation.cc:299] Negotiation successful
0926 03:58:40.331183 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 03:58:40.331450 4811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:40.321559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45331 (local address 127.0.98.2:44989)
0926 03:58:40.321745 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:40.321781 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:40.322308 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:40.322600 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:40.322603 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:40.322933 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:40.323551 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:40.323556 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:40.324359 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:40.324360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:40.325553 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:40.325555 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:40.326085 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:40.326086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:40.326255 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:40.326774 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:40.326784 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:40.328320 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:58:40.329995 (+ 1675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:40.330001 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:40.330003 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:40.330224 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:40.330453 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:40.330453 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:40.330454 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:40.330564 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:40.330818 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:58:40.330823 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:58:40.330962 (+ 139us) client_negotiation.cc:769] Sending connection context
0926 03:58:40.331069 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 03:58:40.331218 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":110,"threads_started":1}
I20260926 03:58:40.331918 4809 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45331
I20260926 03:58:40.331997 4809 heartbeater.cc:464] Registering TS with master...
I20260926 03:58:40.332154 4809 heartbeater.cc:511] Master 127.0.98.62:45331 requested a full tablet report, sending...
I20260926 03:58:40.332597 4486 ts_manager.cc:194] Registered new tserver with Master: 27fb70bd8119450583135702b45e5a38 (127.0.98.2:38299)
I20260926 03:58:40.333025 4486 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44989
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 03:58:40.347596 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45331
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:58:40.426785 4814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:58:40.427016 4814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:58:40.427063 4814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:58:40.428246 4814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:58:40.428311 4814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:58:40.428356 4814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 03:58:40.429864 4814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45331
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:58:40.430135 4814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:58:40.430359 4814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:58:40.435139 4820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:40.435210 4819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:40.435236 4814 server_base.cc:1034] running on GCE node
W20260926 03:58:40.435140 4822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:40.435617 4814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:58:40.435850 4814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:58:40.436995 4814 hybrid_clock.cc:648] HybridClock initialized: now 1790395120436980 us; error 30 us; skew 500 ppm
Sep 26 03:58:40 dist-test-slave-nhpw krb5kdc[4442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395120, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:58:40.439389 4814 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 03:58:40.439739 4814 fs_manager.cc:359] Metadata directory not provided
I20260926 03:58:40.439790 4814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:58:40.439950 4814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:58:40.441030 4814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/minicluster-data/ts-2/data/instance:
uuid: "cd901945dd68454685831c25f235acbd"
format_stamp: "Formatted at 2026-09-26 03:58:40 on dist-test-slave-nhpw"
server_key: "4bb1a99427e569df55a72e4c6cf42e4d"
server_key_iv: "da61fba4cd0fbfc06b885635b115fdab"
server_key_version: "encryptionkey@0"
I20260926 03:58:40.441401 4814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/minicluster-data/ts-2/wal/instance:
uuid: "cd901945dd68454685831c25f235acbd"
format_stamp: "Formatted at 2026-09-26 03:58:40 on dist-test-slave-nhpw"
server_key: "4bb1a99427e569df55a72e4c6cf42e4d"
server_key_iv: "da61fba4cd0fbfc06b885635b115fdab"
server_key_version: "encryptionkey@0"
I20260926 03:58:40.442632 4814 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 03:58:40.443297 4828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:40.443471 4814 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 03:58:40.443567 4814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/minicluster-data/ts-2/wal
uuid: "cd901945dd68454685831c25f235acbd"
format_stamp: "Formatted at 2026-09-26 03:58:40 on dist-test-slave-nhpw"
server_key: "4bb1a99427e569df55a72e4c6cf42e4d"
server_key_iv: "da61fba4cd0fbfc06b885635b115fdab"
server_key_version: "encryptionkey@0"
I20260926 03:58:40.443650 4814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:58:40.459230 4814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:58:40.461061 4814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:58:40.461217 4814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:58:40.461429 4814 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:58:40.461830 4814 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:58:40.461867 4814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:40.461889 4814 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:58:40.461933 4814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:40.467415 4814 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38917
I20260926 03:58:40.467450 4941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38917 every 8 connection(s)
I20260926 03:58:40.467698 4814 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
I20260926 03:58:40.471784 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 4814
I20260926 03:58:40.471891 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/minicluster-data/ts-2/wal/instance
I20260926 03:58:40.472105 392 external_mini_cluster.cc:1428] Setting key 619b83be0dcf43f57f8d046646de0467
Sep 26 03:58:40 dist-test-slave-nhpw krb5kdc[4442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395120, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 03:58:40.476387 4536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:40.468328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41059 (local address 127.0.98.62:45331)
0926 03:58:40.468396 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:40.468399 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:40.468675 (+ 276us) server_negotiation.cc:407] Connection header received
0926 03:58:40.469161 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:40.469162 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:40.469171 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:40.469226 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:40.470086 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:40.470558 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:40.471515 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:40.471697 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:40.471749 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:40.474134 (+ 2385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:40.474136 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:40.474136 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:40.474142 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:40.475304 (+ 1162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:40.475656 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:40.475657 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:40.475657 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:40.475702 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:40.475905 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:40.475906 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:40.475906 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:40.476003 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:40.476056 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:40.476270 (+ 214us) server_negotiation.cc:299] Negotiation successful
0926 03:58:40.476327 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 03:58:40.476550 4944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:40.468323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45331 (local address 127.0.98.3:41059)
0926 03:58:40.468493 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:40.468549 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:40.469063 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:40.469286 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:40.469289 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:40.469542 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:40.470009 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:40.470014 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:40.470638 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:40.470639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:40.471438 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:40.471441 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:40.471779 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:40.471780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:40.471930 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:40.472543 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:40.472553 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:40.474058 (+ 1505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:58:40.475378 (+ 1320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:40.475383 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:40.475385 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:40.475593 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:40.475758 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:40.475759 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:40.475759 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:40.475846 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:40.476056 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:58:40.476060 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:58:40.476162 (+ 102us) client_negotiation.cc:769] Sending connection context
0926 03:58:40.476240 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 03:58:40.476349 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":100,"threads_started":1}
I20260926 03:58:40.477046 4942 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45331
I20260926 03:58:40.477160 4942 heartbeater.cc:464] Registering TS with master...
I20260926 03:58:40.477303 4942 heartbeater.cc:511] Master 127.0.98.62:45331 requested a full tablet report, sending...
I20260926 03:58:40.477726 4486 ts_manager.cc:194] Registered new tserver with Master: cd901945dd68454685831c25f235acbd (127.0.98.3:38917)
I20260926 03:58:40.478132 4486 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41059
I20260926 03:58:40.483762 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:58:40.490576 4536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:40.484983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40706 (local address 127.0.98.62:45331)
0926 03:58:40.485039 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:40.485041 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:40.485179 (+ 138us) server_negotiation.cc:407] Connection header received
0926 03:58:40.485213 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:40.485213 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:40.485221 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:40.485266 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:40.485771 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:40.486222 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:40.486968 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:40.487142 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:40.487194 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:40.488287 (+ 1093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:40.488288 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:40.488288 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:40.488293 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:40.489545 (+ 1252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:40.489848 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:40.489849 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:40.489849 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:40.489909 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:40.490093 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:40.490094 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:40.490095 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:40.490223 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:40.490303 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:40.490404 (+ 101us) server_negotiation.cc:299] Negotiation successful
0926 03:58:40.490461 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 03:58:40.491974 4486 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40706:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:58:40.493566 4486 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:58:40.499766 4953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:40.495636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38299 (local address 127.0.0.1:39974)
0926 03:58:40.495949 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:40.495959 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:40.496011 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:40.496256 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:40.496258 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:40.496260 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:58:40.496340 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:40.496343 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:40.497428 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:40.497429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:40.498691 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:40.498693 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:40.499460 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:40.499461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:40.499545 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:40.499557 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 03:58:40.499637 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 03:58:40.499691 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"mutex_wait_us":3,"thread_start_us":83,"threads_started":1}
I20260926 03:58:40.499848 4954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:40.495632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37905 (local address 127.0.0.1:57098)
0926 03:58:40.495763 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:40.495800 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:40.495866 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:40.496278 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:40.496278 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:40.496281 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:58:40.496340 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:40.496341 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:40.497666 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:40.497667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:40.498708 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:40.498709 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:40.499605 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:40.499606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:40.499664 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:40.499667 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 03:58:40.499717 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 03:58:40.499747 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":49,"threads_started":1}
I20260926 03:58:40.499863 4958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:40.495928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57098 (local address 127.0.98.1:37905)
0926 03:58:40.496103 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:40.496108 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:40.496116 (+ 8us) server_negotiation.cc:407] Connection header received
0926 03:58:40.496157 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:40.496159 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:40.496170 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:40.496265 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:58:40.496431 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:40.497554 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:40.498796 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:40.499476 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:40.499582 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:40.499637 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:40.499728 (+ 91us) server_negotiation.cc:299] Negotiation successful
0926 03:58:40.499774 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":69,"threads_started":1}
I20260926 03:58:40.500066 4955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:40.495726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39974 (local address 127.0.98.2:38299)
0926 03:58:40.496090 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:40.496093 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:40.496103 (+ 10us) server_negotiation.cc:407] Connection header received
0926 03:58:40.496143 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:40.496144 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:40.496157 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:40.496219 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:58:40.496438 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:40.497334 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:40.498780 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:40.499361 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:40.499456 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:40.499515 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:40.499960 (+ 445us) server_negotiation.cc:299] Negotiation successful
0926 03:58:40.500001 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":77,"threads_started":1}
I20260926 03:58:40.500684 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:40.495822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38917 (local address 127.0.0.1:45346)
0926 03:58:40.496363 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:40.496373 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:40.496421 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:40.496791 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:40.496792 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:40.496794 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:58:40.496834 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:40.496836 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:40.497953 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:40.497954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:40.498928 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:40.498931 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:40.500419 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:40.500420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:40.500481 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:40.500485 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 03:58:40.500552 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 03:58:40.500591 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"mutex_wait_us":21,"thread_start_us":83,"threads_started":1}
I20260926 03:58:40.500566 4610 tablet_service.cc:1467] Processing CreateTablet for tablet af2c45d26ff0487e90bf10a28c1bfeb7 (DEFAULT_TABLE table=test-table [id=f9add1a4b1e24be2b671acaecb49ddc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:58:40.500727 4743 tablet_service.cc:1467] Processing CreateTablet for tablet af2c45d26ff0487e90bf10a28c1bfeb7 (DEFAULT_TABLE table=test-table [id=f9add1a4b1e24be2b671acaecb49ddc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:58:40.500936 4610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af2c45d26ff0487e90bf10a28c1bfeb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:58:40.500972 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:40.495907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45346 (local address 127.0.98.3:38917)
0926 03:58:40.496037 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:40.496042 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:40.496546 (+ 504us) server_negotiation.cc:407] Connection header received
0926 03:58:40.496585 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:40.496587 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:40.496601 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:40.496725 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:58:40.496916 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:40.497862 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:40.499021 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:40.499704 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:40.499766 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:40.499810 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:40.500857 (+ 1047us) server_negotiation.cc:299] Negotiation successful
0926 03:58:40.500899 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":74,"threads_started":1}
I20260926 03:58:40.501056 4743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af2c45d26ff0487e90bf10a28c1bfeb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:58:40.501573 4876 tablet_service.cc:1467] Processing CreateTablet for tablet af2c45d26ff0487e90bf10a28c1bfeb7 (DEFAULT_TABLE table=test-table [id=f9add1a4b1e24be2b671acaecb49ddc8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:58:40.501842 4876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af2c45d26ff0487e90bf10a28c1bfeb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:58:40.503291 4959 tablet_bootstrap.cc:492] T af2c45d26ff0487e90bf10a28c1bfeb7 P c4e698cb5e63464c926f0c3c6ecf508c: Bootstrap starting.
I20260926 03:58:40.504179 4959 tablet_bootstrap.cc:654] T af2c45d26ff0487e90bf10a28c1bfeb7 P c4e698cb5e63464c926f0c3c6ecf508c: Neither blocks nor log segments found. Creating new log.
I20260926 03:58:40.504283 4960 tablet_bootstrap.cc:492] T af2c45d26ff0487e90bf10a28c1bfeb7 P 27fb70bd8119450583135702b45e5a38: Bootstrap starting.
I20260926 03:58:40.504832 4959 log.cc:824] T af2c45d26ff0487e90bf10a28c1bfeb7 P c4e698cb5e63464c926f0c3c6ecf508c: Log is configured to *not* fsync() on all Append() calls
I20260926 03:58:40.504941 4961 tablet_bootstrap.cc:492] T af2c45d26ff0487e90bf10a28c1bfeb7 P cd901945dd68454685831c25f235acbd: Bootstrap starting.
I20260926 03:58:40.505141 4960 tablet_bootstrap.cc:654] T af2c45d26ff0487e90bf10a28c1bfeb7 P 27fb70bd8119450583135702b45e5a38: Neither blocks nor log segments found. Creating new log.
I20260926 03:58:40.505544 4961 tablet_bootstrap.cc:654] T af2c45d26ff0487e90bf10a28c1bfeb7 P cd901945dd68454685831c25f235acbd: Neither blocks nor log segments found. Creating new log.
I20260926 03:58:40.505613 4960 log.cc:824] T af2c45d26ff0487e90bf10a28c1bfeb7 P 27fb70bd8119450583135702b45e5a38: Log is configured to *not* fsync() on all Append() calls
I20260926 03:58:40.506049 4961 log.cc:824] T af2c45d26ff0487e90bf10a28c1bfeb7 P cd901945dd68454685831c25f235acbd: Log is configured to *not* fsync() on all Append() calls
I20260926 03:58:40.506386 4960 tablet_bootstrap.cc:492] T af2c45d26ff0487e90bf10a28c1bfeb7 P 27fb70bd8119450583135702b45e5a38: No bootstrap required, opened a new log
I20260926 03:58:40.506465 4960 ts_tablet_manager.cc:1397] T af2c45d26ff0487e90bf10a28c1bfeb7 P 27fb70bd8119450583135702b45e5a38: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 03:58:40.506758 4961 tablet_bootstrap.cc:492] T af2c45d26ff0487e90bf10a28c1bfeb7 P cd901945dd68454685831c25f235acbd: No bootstrap required, opened a new log
I20260926 03:58:40.506821 4961 ts_tablet_manager.cc:1397] T af2c45d26ff0487e90bf10a28c1bfeb7 P cd901945dd68454685831c25f235acbd: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 03:58:40.508440 4960 raft_consensus.cc:348] T af2c45d26ff0487e90bf10a28c1bfeb7 P 27fb70bd8119450583135702b45e5a38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27fb70bd8119450583135702b45e5a38" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38299 } } peers { permanent_uuid: "c4e698cb5e63464c926f0c3c6ecf508c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37905 } } peers { permanent_uuid: "cd901945dd68454685831c25f235acbd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38917 } }
I20260926 03:58:40.508625 4960 raft_consensus.cc:374] T af2c45d26ff0487e90bf10a28c1bfeb7 P 27fb70bd8119450583135702b45e5a38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:58:40.508688 4960 raft_consensus.cc:729] T af2c45d26ff0487e90bf10a28c1bfeb7 P 27fb70bd8119450583135702b45e5a38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27fb70bd8119450583135702b45e5a38, State: Initialized, Role: FOLLOWER
I20260926 03:58:40.508705 4961 raft_consensus.cc:348] T af2c45d26ff0487e90bf10a28c1bfeb7 P cd901945dd68454685831c25f235acbd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27fb70bd8119450583135702b45e5a38" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38299 } } peers { permanent_uuid: "c4e698cb5e63464c926f0c3c6ecf508c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37905 } } peers { permanent_uuid: "cd901945dd68454685831c25f235acbd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38917 } }
I20260926 03:58:40.508805 4960 consensus_queue.cc:260] T af2c45d26ff0487e90bf10a28c1bfeb7 P 27fb70bd8119450583135702b45e5a38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27fb70bd8119450583135702b45e5a38" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38299 } } peers { permanent_uuid: "c4e698cb5e63464c926f0c3c6ecf508c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37905 } } peers { permanent_uuid: "cd901945dd68454685831c25f235acbd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38917 } }
I20260926 03:58:40.508846 4961 raft_consensus.cc:374] T af2c45d26ff0487e90bf10a28c1bfeb7 P cd901945dd68454685831c25f235acbd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:58:40.508880 4961 raft_consensus.cc:729] T af2c45d26ff0487e90bf10a28c1bfeb7 P cd901945dd68454685831c25f235acbd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd901945dd68454685831c25f235acbd, State: Initialized, Role: FOLLOWER
I20260926 03:58:40.508987 4959 tablet_bootstrap.cc:492] T af2c45d26ff0487e90bf10a28c1bfeb7 P c4e698cb5e63464c926f0c3c6ecf508c: No bootstrap required, opened a new log
I20260926 03:58:40.509006 4961 consensus_queue.cc:260] T af2c45d26ff0487e90bf10a28c1bfeb7 P cd901945dd68454685831c25f235acbd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27fb70bd8119450583135702b45e5a38" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38299 } } peers { permanent_uuid: "c4e698cb5e63464c926f0c3c6ecf508c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37905 } } peers { permanent_uuid: "cd901945dd68454685831c25f235acbd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38917 } }
I20260926 03:58:40.509058 4959 ts_tablet_manager.cc:1397] T af2c45d26ff0487e90bf10a28c1bfeb7 P c4e698cb5e63464c926f0c3c6ecf508c: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260926 03:58:40.509073 4960 ts_tablet_manager.cc:1428] T af2c45d26ff0487e90bf10a28c1bfeb7 P 27fb70bd8119450583135702b45e5a38: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 03:58:40.509112 4809 heartbeater.cc:503] Master 127.0.98.62:45331 was elected leader, sending a full tablet report...
I20260926 03:58:40.509300 4942 heartbeater.cc:503] Master 127.0.98.62:45331 was elected leader, sending a full tablet report...
I20260926 03:58:40.509284 4961 ts_tablet_manager.cc:1428] T af2c45d26ff0487e90bf10a28c1bfeb7 P cd901945dd68454685831c25f235acbd: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 03:58:40.511056 4959 raft_consensus.cc:348] T af2c45d26ff0487e90bf10a28c1bfeb7 P c4e698cb5e63464c926f0c3c6ecf508c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27fb70bd8119450583135702b45e5a38" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38299 } } peers { permanent_uuid: "c4e698cb5e63464c926f0c3c6ecf508c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37905 } } peers { permanent_uuid: "cd901945dd68454685831c25f235acbd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38917 } }
I20260926 03:58:40.511216 4959 raft_consensus.cc:374] T af2c45d26ff0487e90bf10a28c1bfeb7 P c4e698cb5e63464c926f0c3c6ecf508c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:58:40.511264 4959 raft_consensus.cc:729] T af2c45d26ff0487e90bf10a28c1bfeb7 P c4e698cb5e63464c926f0c3c6ecf508c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4e698cb5e63464c926f0c3c6ecf508c, State: Initialized, Role: FOLLOWER
I20260926 03:58:40.511374 4959 consensus_queue.cc:260] T af2c45d26ff0487e90bf10a28c1bfeb7 P c4e698cb5e63464c926f0c3c6ecf508c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27fb70bd8119450583135702b45e5a38" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38299 } } peers { permanent_uuid: "c4e698cb5e63464c926f0c3c6ecf508c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37905 } } peers { permanent_uuid: "cd901945dd68454685831c25f235acbd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38917 } }
I20260926 03:58:40.511626 4959 ts_tablet_manager.cc:1428] T af2c45d26ff0487e90bf10a28c1bfeb7 P c4e698cb5e63464c926f0c3c6ecf508c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 03:58:40.511652 4676 heartbeater.cc:503] Master 127.0.98.62:45331 was elected leader, sending a full tablet report...
W20260926 03:58:40.571357 4810 tablet.cc:2369] T af2c45d26ff0487e90bf10a28c1bfeb7 P 27fb70bd8119450583135702b45e5a38: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:58:40.606997 4967 raft_consensus.cc:482] T af2c45d26ff0487e90bf10a28c1bfeb7 P c4e698cb5e63464c926f0c3c6ecf508c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:58:40.607161 4967 raft_consensus.cc:504] T af2c45d26ff0487e90bf10a28c1bfeb7 P c4e698cb5e63464c926f0c3c6ecf508c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27fb70bd8119450583135702b45e5a38" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38299 } } peers { permanent_uuid: "c4e698cb5e63464c926f0c3c6ecf508c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37905 } } peers { permanent_uuid: "cd901945dd68454685831c25f235acbd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38917 } }
I20260926 03:58:40.607566 4967 leader_election.cc:288] T af2c45d26ff0487e90bf10a28c1bfeb7 P c4e698cb5e63464c926f0c3c6ecf508c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 27fb70bd8119450583135702b45e5a38 (127.0.98.2:38299), cd901945dd68454685831c25f235acbd (127.0.98.3:38917)
I20260926 03:58:40.610595 4955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:40.607827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33079 (local address 127.0.98.2:38299)
0926 03:58:40.607874 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:40.607876 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:40.608074 (+ 198us) server_negotiation.cc:407] Connection header received
0926 03:58:40.608107 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:40.608107 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:40.608118 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:40.608163 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:58:40.608343 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:40.608876 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:40.609897 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:40.610281 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:40.610336 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:40.610344 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:40.610490 (+ 146us) server_negotiation.cc:299] Negotiation successful
0926 03:58:40.610528 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 03:58:40.610641 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:40.607900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50463 (local address 127.0.98.3:38917)
0926 03:58:40.607943 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:40.607945 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:40.607954 (+ 9us) server_negotiation.cc:407] Connection header received
0926 03:58:40.607984 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:40.607985 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:40.607998 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:40.608071 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:58:40.608343 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:40.608880 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:40.609867 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:40.610329 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:40.610401 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:40.610410 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:40.610535 (+ 125us) server_negotiation.cc:299] Negotiation successful
0926 03:58:40.610575 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 03:58:40.610642 4678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:40.607756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38917 (local address 127.0.98.1:50463)
0926 03:58:40.607812 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:40.607824 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:40.607867 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:40.608206 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:40.608207 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:40.608208 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:58:40.608257 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:40.608258 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:40.608960 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:40.608961 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:40.609779 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:40.609782 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:40.610396 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:40.610397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:40.610448 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:40.610451 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 03:58:40.610513 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 03:58:40.610557 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":2}
I20260926 03:58:40.610641 4968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:40.607756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38299 (local address 127.0.98.1:33079)
0926 03:58:40.607986 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:40.607996 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:40.608062 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:40.608184 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:40.608185 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:40.608187 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:58:40.608250 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:40.608252 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:40.608965 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:40.608966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:40.609809 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:40.609811 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:40.610356 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:40.610357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:40.610428 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:40.610432 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 03:58:40.610483 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 03:58:40.610521 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"mutex_wait_us":11,"thread_start_us":120,"threads_started":1}
I20260926 03:58:40.611084 4763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af2c45d26ff0487e90bf10a28c1bfeb7" candidate_uuid: "c4e698cb5e63464c926f0c3c6ecf508c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27fb70bd8119450583135702b45e5a38" is_pre_election: true
I20260926 03:58:40.611282 4763 raft_consensus.cc:2454] T af2c45d26ff0487e90bf10a28c1bfeb7 P 27fb70bd8119450583135702b45e5a38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4e698cb5e63464c926f0c3c6ecf508c in term 0.
I20260926 03:58:40.611246 4896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af2c45d26ff0487e90bf10a28c1bfeb7" candidate_uuid: "c4e698cb5e63464c926f0c3c6ecf508c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd901945dd68454685831c25f235acbd" is_pre_election: true
I20260926 03:58:40.611369 4896 raft_consensus.cc:2454] T af2c45d26ff0487e90bf10a28c1bfeb7 P cd901945dd68454685831c25f235acbd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4e698cb5e63464c926f0c3c6ecf508c in term 0.
I20260926 03:58:40.611497 4563 leader_election.cc:302] T af2c45d26ff0487e90bf10a28c1bfeb7 P c4e698cb5e63464c926f0c3c6ecf508c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27fb70bd8119450583135702b45e5a38, c4e698cb5e63464c926f0c3c6ecf508c; no voters:
I20260926 03:58:40.611626 4967 raft_consensus.cc:2781] T af2c45d26ff0487e90bf10a28c1bfeb7 P c4e698cb5e63464c926f0c3c6ecf508c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:58:40.611680 4967 raft_consensus.cc:482] T af2c45d26ff0487e90bf10a28c1bfeb7 P c4e698cb5e63464c926f0c3c6ecf508c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:58:40.611701 4967 raft_consensus.cc:3037] T af2c45d26ff0487e90bf10a28c1bfeb7 P c4e698cb5e63464c926f0c3c6ecf508c [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:58:40.612372 4967 raft_consensus.cc:504] T af2c45d26ff0487e90bf10a28c1bfeb7 P c4e698cb5e63464c926f0c3c6ecf508c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27fb70bd8119450583135702b45e5a38" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38299 } } peers { permanent_uuid: "c4e698cb5e63464c926f0c3c6ecf508c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37905 } } peers { permanent_uuid: "cd901945dd68454685831c25f235acbd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38917 } }
I20260926 03:58:40.612505 4967 leader_election.cc:288] T af2c45d26ff0487e90bf10a28c1bfeb7 P c4e698cb5e63464c926f0c3c6ecf508c [CANDIDATE]: Term 1 election: Requested vote from peers 27fb70bd8119450583135702b45e5a38 (127.0.98.2:38299), cd901945dd68454685831c25f235acbd (127.0.98.3:38917)
I20260926 03:58:40.612675 4896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af2c45d26ff0487e90bf10a28c1bfeb7" candidate_uuid: "c4e698cb5e63464c926f0c3c6ecf508c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd901945dd68454685831c25f235acbd"
I20260926 03:58:40.612687 4763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af2c45d26ff0487e90bf10a28c1bfeb7" candidate_uuid: "c4e698cb5e63464c926f0c3c6ecf508c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "27fb70bd8119450583135702b45e5a38"
I20260926 03:58:40.612761 4763 raft_consensus.cc:3037] T af2c45d26ff0487e90bf10a28c1bfeb7 P 27fb70bd8119450583135702b45e5a38 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:58:40.612768 4896 raft_consensus.cc:3037] T af2c45d26ff0487e90bf10a28c1bfeb7 P cd901945dd68454685831c25f235acbd [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:58:40.613472 4763 raft_consensus.cc:2454] T af2c45d26ff0487e90bf10a28c1bfeb7 P 27fb70bd8119450583135702b45e5a38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4e698cb5e63464c926f0c3c6ecf508c in term 1.
I20260926 03:58:40.613484 4896 raft_consensus.cc:2454] T af2c45d26ff0487e90bf10a28c1bfeb7 P cd901945dd68454685831c25f235acbd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4e698cb5e63464c926f0c3c6ecf508c in term 1.
I20260926 03:58:40.613631 4563 leader_election.cc:302] T af2c45d26ff0487e90bf10a28c1bfeb7 P c4e698cb5e63464c926f0c3c6ecf508c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27fb70bd8119450583135702b45e5a38, c4e698cb5e63464c926f0c3c6ecf508c; no voters:
I20260926 03:58:40.613722 4967 raft_consensus.cc:2781] T af2c45d26ff0487e90bf10a28c1bfeb7 P c4e698cb5e63464c926f0c3c6ecf508c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:58:40.613865 4967 raft_consensus.cc:686] T af2c45d26ff0487e90bf10a28c1bfeb7 P c4e698cb5e63464c926f0c3c6ecf508c [term 1 LEADER]: Becoming Leader. State: Replica: c4e698cb5e63464c926f0c3c6ecf508c, State: Running, Role: LEADER
I20260926 03:58:40.613963 4967 consensus_queue.cc:237] T af2c45d26ff0487e90bf10a28c1bfeb7 P c4e698cb5e63464c926f0c3c6ecf508c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27fb70bd8119450583135702b45e5a38" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38299 } } peers { permanent_uuid: "c4e698cb5e63464c926f0c3c6ecf508c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37905 } } peers { permanent_uuid: "cd901945dd68454685831c25f235acbd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38917 } }
I20260926 03:58:40.614751 4486 catalog_manager.cc:5495] T af2c45d26ff0487e90bf10a28c1bfeb7 P c4e698cb5e63464c926f0c3c6ecf508c reported cstate change: term changed from 0 to 1, leader changed from <none> to c4e698cb5e63464c926f0c3c6ecf508c (127.0.98.1). New cstate: current_term: 1 leader_uuid: "c4e698cb5e63464c926f0c3c6ecf508c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "27fb70bd8119450583135702b45e5a38" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c4e698cb5e63464c926f0c3c6ecf508c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd901945dd68454685831c25f235acbd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38917 } health_report { overall_health: UNKNOWN } } }
I20260926 03:58:40.642748 4958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:40.640413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57100 (local address 127.0.98.1:37905)
0926 03:58:40.640457 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:40.640459 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:40.640468 (+ 9us) server_negotiation.cc:407] Connection header received
0926 03:58:40.640540 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:40.640541 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:40.640558 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:40.640634 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:58:40.640776 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:40.641218 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:40.641958 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:40.642148 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:40.642247 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:40.642414 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:58:40.642512 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:58:40.642612 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:40.642631 (+ 19us) server_negotiation.cc:299] Negotiation successful
0926 03:58:40.642649 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 03:58:40.644402 4896 raft_consensus.cc:1261] T af2c45d26ff0487e90bf10a28c1bfeb7 P cd901945dd68454685831c25f235acbd [term 1 FOLLOWER]: Refusing update from remote peer c4e698cb5e63464c926f0c3c6ecf508c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:58:40.644388 4763 raft_consensus.cc:1261] T af2c45d26ff0487e90bf10a28c1bfeb7 P 27fb70bd8119450583135702b45e5a38 [term 1 FOLLOWER]: Refusing update from remote peer c4e698cb5e63464c926f0c3c6ecf508c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:58:40.644717 4969 consensus_queue.cc:1035] T af2c45d26ff0487e90bf10a28c1bfeb7 P c4e698cb5e63464c926f0c3c6ecf508c [LEADER]: Connected to new peer: Peer: permanent_uuid: "27fb70bd8119450583135702b45e5a38" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:58:40.644836 4967 consensus_queue.cc:1035] T af2c45d26ff0487e90bf10a28c1bfeb7 P c4e698cb5e63464c926f0c3c6ecf508c [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd901945dd68454685831c25f235acbd" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:58:40.647068 4972 mvcc.cc:205] Tried to move back new op lower bound from 7333458414156636160 to 7333458414035533824. Current Snapshot: MvccSnapshot[applied={T|T < 7333458414156636160}]
I20260926 03:58:40.647241 4974 mvcc.cc:205] Tried to move back new op lower bound from 7333458414156636160 to 7333458414035533824. Current Snapshot: MvccSnapshot[applied={T|T < 7333458414156636160}]
I20260926 03:58:40.647998 4973 mvcc.cc:205] Tried to move back new op lower bound from 7333458414156636160 to 7333458414035533824. Current Snapshot: MvccSnapshot[applied={T|T < 7333458414156636160}]
I20260926 03:58:40.649577 4486 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:58:40.649679 4486 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:58:40.650348 4486 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 17e0a77798b844359665d39d3e99903c: Sending DeleteTablet for 3 replicas of tablet af2c45d26ff0487e90bf10a28c1bfeb7
I20260926 03:58:40.650632 4743 tablet_service.cc:1514] Processing DeleteTablet for tablet af2c45d26ff0487e90bf10a28c1bfeb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:58:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39974
I20260926 03:58:40.650681 4876 tablet_service.cc:1514] Processing DeleteTablet for tablet af2c45d26ff0487e90bf10a28c1bfeb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:58:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45346
I20260926 03:58:40.650704 4610 tablet_service.cc:1514] Processing DeleteTablet for tablet af2c45d26ff0487e90bf10a28c1bfeb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:58:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57098
I20260926 03:58:40.650879 4982 tablet_replica.cc:330] T af2c45d26ff0487e90bf10a28c1bfeb7 P cd901945dd68454685831c25f235acbd: stopping tablet replica
I20260926 03:58:40.650970 4982 raft_consensus.cc:2229] T af2c45d26ff0487e90bf10a28c1bfeb7 P cd901945dd68454685831c25f235acbd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:58:40.651087 4981 tablet_replica.cc:330] T af2c45d26ff0487e90bf10a28c1bfeb7 P 27fb70bd8119450583135702b45e5a38: stopping tablet replica
I20260926 03:58:40.651113 4982 raft_consensus.cc:2258] T af2c45d26ff0487e90bf10a28c1bfeb7 P cd901945dd68454685831c25f235acbd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:58:40.651178 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 4548
I20260926 03:58:40.651249 4983 tablet_replica.cc:330] T af2c45d26ff0487e90bf10a28c1bfeb7 P c4e698cb5e63464c926f0c3c6ecf508c: stopping tablet replica
I20260926 03:58:40.651270 4981 raft_consensus.cc:2229] T af2c45d26ff0487e90bf10a28c1bfeb7 P 27fb70bd8119450583135702b45e5a38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:58:40.651337 4983 raft_consensus.cc:2229] T af2c45d26ff0487e90bf10a28c1bfeb7 P c4e698cb5e63464c926f0c3c6ecf508c [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:58:40.651423 4983 raft_consensus.cc:2258] T af2c45d26ff0487e90bf10a28c1bfeb7 P c4e698cb5e63464c926f0c3c6ecf508c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:58:40.653323 4981 raft_consensus.cc:2258] T af2c45d26ff0487e90bf10a28c1bfeb7 P 27fb70bd8119450583135702b45e5a38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:58:40.653754 4981 ts_tablet_manager.cc:1905] T af2c45d26ff0487e90bf10a28c1bfeb7 P 27fb70bd8119450583135702b45e5a38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:58:40.655666 4981 ts_tablet_manager.cc:1918] T af2c45d26ff0487e90bf10a28c1bfeb7 P 27fb70bd8119450583135702b45e5a38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:58:40.655740 4981 log.cc:1196] T af2c45d26ff0487e90bf10a28c1bfeb7 P 27fb70bd8119450583135702b45e5a38: Deleting WAL directory at /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/minicluster-data/ts-1/wal/wals/af2c45d26ff0487e90bf10a28c1bfeb7
I20260926 03:58:40.656018 4981 ts_tablet_manager.cc:1939] T af2c45d26ff0487e90bf10a28c1bfeb7 P 27fb70bd8119450583135702b45e5a38: Deleting consensus metadata
I20260926 03:58:40.656106 4982 ts_tablet_manager.cc:1905] T af2c45d26ff0487e90bf10a28c1bfeb7 P cd901945dd68454685831c25f235acbd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:58:40.656534 4470 catalog_manager.cc:4841] TS 27fb70bd8119450583135702b45e5a38 (127.0.98.2:38299): tablet af2c45d26ff0487e90bf10a28c1bfeb7 (table test-table [id=f9add1a4b1e24be2b671acaecb49ddc8]) successfully deleted
W20260926 03:58:40.657317 4473 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37905 (error 108)
I20260926 03:58:40.657809 4982 ts_tablet_manager.cc:1918] T af2c45d26ff0487e90bf10a28c1bfeb7 P cd901945dd68454685831c25f235acbd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:58:40.657841 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 4681
I20260926 03:58:40.657881 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:40.657733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37905 (local address 127.0.0.1:57102)
0926 03:58:40.657790 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:40.657823 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 03:58:40.657882 4982 log.cc:1196] T af2c45d26ff0487e90bf10a28c1bfeb7 P cd901945dd68454685831c25f235acbd: Deleting WAL directory at /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790395095474643-392-0/minicluster-data/ts-2/wal/wals/af2c45d26ff0487e90bf10a28c1bfeb7
W20260926 03:58:40.658084 4473 catalog_manager.cc:4584] TS c4e698cb5e63464c926f0c3c6ecf508c (127.0.98.1:37905): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet af2c45d26ff0487e90bf10a28c1bfeb7: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37905: connect: Connection refused (error 111)
I20260926 03:58:40.658160 4982 ts_tablet_manager.cc:1939] T af2c45d26ff0487e90bf10a28c1bfeb7 P cd901945dd68454685831c25f235acbd: Deleting consensus metadata
I20260926 03:58:40.658528 4473 catalog_manager.cc:4841] TS cd901945dd68454685831c25f235acbd (127.0.98.3:38917): tablet af2c45d26ff0487e90bf10a28c1bfeb7 (table test-table [id=f9add1a4b1e24be2b671acaecb49ddc8]) successfully deleted
I20260926 03:58:40.663339 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 4814
I20260926 03:58:40.667529 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 4455
2026-09-26T03:58:40Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1997 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-26T03:58:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:58:40Z Disabled control of system clock
I20260926 03:58:40.694952 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395095474643-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395095474643-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395095474643-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45093
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45093
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 03:58:40.773842 4988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:58:40.774030 4988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:58:40.774051 4988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:58:40.775177 4988 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:58:40.775218 4988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:58:40.775230 4988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:58:40.775242 4988 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:58:40.775295 4988 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:58:40.776613 4988 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395095474643-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395095474643-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45093
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4988
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395095474643-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:58:40.776871 4988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:58:40.777159 4988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:58:40.781888 4996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:40.781924 4994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:40.781927 4993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:40.781945 4988 server_base.cc:1034] running on GCE node
I20260926 03:58:40.782369 4988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:58:40.782684 4988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:58:40.783836 4988 hybrid_clock.cc:648] HybridClock initialized: now 1790395120783818 us; error 55 us; skew 500 ppm
I20260926 03:58:40.785032 4988 webserver.cc:460] Webserver started at http://127.0.98.62:44329/ using document root <none> and password file <none>
I20260926 03:58:40.785269 4988 fs_manager.cc:359] Metadata directory not provided
I20260926 03:58:40.785346 4988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:58:40.785495 4988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:58:40.786469 4988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395095474643-392-0/minicluster-data/master-0/data/instance:
uuid: "c68f8cac93a84d68bdd27e505c5ec226"
format_stamp: "Formatted at 2026-09-26 03:58:40 on dist-test-slave-nhpw"
server_key: "563a2526dfc54090e5c049daa778b3e3"
server_key_iv: "e8b2bcb84b6fe807c527f7603e36c441"
server_key_version: "encryptionkey@0"
I20260926 03:58:40.786813 4988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395095474643-392-0/minicluster-data/master-0/wal/instance:
uuid: "c68f8cac93a84d68bdd27e505c5ec226"
format_stamp: "Formatted at 2026-09-26 03:58:40 on dist-test-slave-nhpw"
server_key: "563a2526dfc54090e5c049daa778b3e3"
server_key_iv: "e8b2bcb84b6fe807c527f7603e36c441"
server_key_version: "encryptionkey@0"
I20260926 03:58:40.788040 4988 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 03:58:40.788762 5002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:40.788942 4988 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:40.789045 4988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395095474643-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395095474643-392-0/minicluster-data/master-0/wal
uuid: "c68f8cac93a84d68bdd27e505c5ec226"
format_stamp: "Formatted at 2026-09-26 03:58:40 on dist-test-slave-nhpw"
server_key: "563a2526dfc54090e5c049daa778b3e3"
server_key_iv: "e8b2bcb84b6fe807c527f7603e36c441"
server_key_version: "encryptionkey@0"
I20260926 03:58:40.789134 4988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395095474643-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395095474643-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395095474643-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:58:40.792295 4988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:58:40.792552 4988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:58:40.792719 4988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:58:40.796501 4988 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45093
I20260926 03:58:40.796535 5054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45093 every 8 connection(s)
I20260926 03:58:40.796870 4988 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
I20260926 03:58:40.797565 5055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:58:40.799122 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 4988
I20260926 03:58:40.799233 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395095474643-392-0/minicluster-data/master-0/wal/instance
I20260926 03:58:40.799403 392 external_mini_cluster.cc:1428] Setting key 7c100f0cf5ef6abacfea63f08d5299c9
I20260926 03:58:40.800545 5055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c68f8cac93a84d68bdd27e505c5ec226: Bootstrap starting.
I20260926 03:58:40.801260 5055 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c68f8cac93a84d68bdd27e505c5ec226: Neither blocks nor log segments found. Creating new log.
I20260926 03:58:40.801589 5055 log.cc:824] T 00000000000000000000000000000000 P c68f8cac93a84d68bdd27e505c5ec226: Log is configured to *not* fsync() on all Append() calls
I20260926 03:58:40.802119 5058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:40.799932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49246 (local address 127.0.98.62:45093)
0926 03:58:40.800131 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:40.800141 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:40.800174 (+ 33us) server_negotiation.cc:407] Connection header received
0926 03:58:40.800742 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:40.800750 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:40.800800 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:40.800952 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:40.801588 (+ 636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:40.801591 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:40.801592 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:58:40.801597 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:40.801641 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:58:40.801655 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:40.801752 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:40.801852 (+ 100us) server_negotiation.cc:299] Negotiation successful
0926 03:58:40.801930 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":75,"threads_started":1}
I20260926 03:58:40.802321 5055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c68f8cac93a84d68bdd27e505c5ec226: No bootstrap required, opened a new log
I20260926 03:58:40.803706 5055 raft_consensus.cc:348] T 00000000000000000000000000000000 P c68f8cac93a84d68bdd27e505c5ec226 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c68f8cac93a84d68bdd27e505c5ec226" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45093 } }
I20260926 03:58:40.803831 5055 raft_consensus.cc:374] T 00000000000000000000000000000000 P c68f8cac93a84d68bdd27e505c5ec226 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:58:40.803854 5055 raft_consensus.cc:729] T 00000000000000000000000000000000 P c68f8cac93a84d68bdd27e505c5ec226 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c68f8cac93a84d68bdd27e505c5ec226, State: Initialized, Role: FOLLOWER
I20260926 03:58:40.803993 5055 consensus_queue.cc:260] T 00000000000000000000000000000000 P c68f8cac93a84d68bdd27e505c5ec226 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c68f8cac93a84d68bdd27e505c5ec226" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45093 } }
I20260926 03:58:40.804087 5055 raft_consensus.cc:388] T 00000000000000000000000000000000 P c68f8cac93a84d68bdd27e505c5ec226 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:58:40.804132 5055 raft_consensus.cc:482] T 00000000000000000000000000000000 P c68f8cac93a84d68bdd27e505c5ec226 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:58:40.804181 5055 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c68f8cac93a84d68bdd27e505c5ec226 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:58:40.805001 5055 raft_consensus.cc:504] T 00000000000000000000000000000000 P c68f8cac93a84d68bdd27e505c5ec226 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c68f8cac93a84d68bdd27e505c5ec226" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45093 } }
I20260926 03:58:40.805090 5055 leader_election.cc:302] T 00000000000000000000000000000000 P c68f8cac93a84d68bdd27e505c5ec226 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c68f8cac93a84d68bdd27e505c5ec226; no voters:
I20260926 03:58:40.805272 5055 leader_election.cc:288] T 00000000000000000000000000000000 P c68f8cac93a84d68bdd27e505c5ec226 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:58:40.805303 5060 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c68f8cac93a84d68bdd27e505c5ec226 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:58:40.805411 5060 raft_consensus.cc:686] T 00000000000000000000000000000000 P c68f8cac93a84d68bdd27e505c5ec226 [term 1 LEADER]: Becoming Leader. State: Replica: c68f8cac93a84d68bdd27e505c5ec226, State: Running, Role: LEADER
I20260926 03:58:40.805492 5060 consensus_queue.cc:237] T 00000000000000000000000000000000 P c68f8cac93a84d68bdd27e505c5ec226 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c68f8cac93a84d68bdd27e505c5ec226" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45093 } }
I20260926 03:58:40.805547 5055 sys_catalog.cc:564] T 00000000000000000000000000000000 P c68f8cac93a84d68bdd27e505c5ec226 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:58:40.805815 5061 sys_catalog.cc:455] T 00000000000000000000000000000000 P c68f8cac93a84d68bdd27e505c5ec226 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c68f8cac93a84d68bdd27e505c5ec226. Latest consensus state: current_term: 1 leader_uuid: "c68f8cac93a84d68bdd27e505c5ec226" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c68f8cac93a84d68bdd27e505c5ec226" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45093 } } }
I20260926 03:58:40.805910 5061 sys_catalog.cc:458] T 00000000000000000000000000000000 P c68f8cac93a84d68bdd27e505c5ec226 [sys.catalog]: This master's current role is: LEADER
I20260926 03:58:40.806061 5062 sys_catalog.cc:455] T 00000000000000000000000000000000 P c68f8cac93a84d68bdd27e505c5ec226 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c68f8cac93a84d68bdd27e505c5ec226" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c68f8cac93a84d68bdd27e505c5ec226" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45093 } } }
I20260926 03:58:40.806133 5062 sys_catalog.cc:458] T 00000000000000000000000000000000 P c68f8cac93a84d68bdd27e505c5ec226 [sys.catalog]: This master's current role is: LEADER
I20260926 03:58:40.806411 5065 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:58:40.806869 5065 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:58:40.808243 5065 catalog_manager.cc:1347] Generated new cluster ID: 4afa58f0b1ec41b196b5bfb345d2c79c
I20260926 03:58:40.808295 5065 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:58:40.825668 5065 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:58:40.825826 5065 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:58:40.836117 5065 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c68f8cac93a84d68bdd27e505c5ec226: Generated new TSK 0
I20260926 03:58:40.836315 5065 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:58:40.839859 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395095474643-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395095474643-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395095474643-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45093
--builtin_ntp_servers=127.0.98.20:35287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 03:58:40.929576 5079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:58:40.929797 5079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:58:40.929831 5079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:58:40.931564 5079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:58:40.931620 5079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:58:40.931661 5079 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 03:58:40.932996 5079 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395095474643-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395095474643-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5079
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395095474643-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:58:40.933245 5079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:58:40.933534 5079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:58:40.938544 5084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:40.938647 5085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:40.938758 5079 server_base.cc:1034] running on GCE node
W20260926 03:58:40.938551 5087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:40.939035 5079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:58:40.939296 5079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:58:40.940449 5079 hybrid_clock.cc:648] HybridClock initialized: now 1790395120940433 us; error 31 us; skew 500 ppm
I20260926 03:58:40.941828 5079 webserver.cc:460] Webserver started at http://127.0.98.1:45695/ using document root <none> and password file <none>
I20260926 03:58:40.942080 5079 fs_manager.cc:359] Metadata directory not provided
I20260926 03:58:40.942145 5079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:58:40.942272 5079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:58:40.943627 5079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395095474643-392-0/minicluster-data/ts-0/data/instance:
uuid: "3ec23d7a3be648f4b233e06efd4821dd"
format_stamp: "Formatted at 2026-09-26 03:58:40 on dist-test-slave-nhpw"
server_key: "752d55c54a21e00d6c81c77012ff0c1b"
server_key_iv: "ae8141fd35f496927b5bdaf37acb96a3"
server_key_version: "encryptionkey@0"
I20260926 03:58:40.944016 5079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395095474643-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3ec23d7a3be648f4b233e06efd4821dd"
format_stamp: "Formatted at 2026-09-26 03:58:40 on dist-test-slave-nhpw"
server_key: "752d55c54a21e00d6c81c77012ff0c1b"
server_key_iv: "ae8141fd35f496927b5bdaf37acb96a3"
server_key_version: "encryptionkey@0"
I20260926 03:58:40.945683 5079 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260926 03:58:40.946723 5093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:40.946902 5079 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 03:58:40.946990 5079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395095474643-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395095474643-392-0/minicluster-data/ts-0/wal
uuid: "3ec23d7a3be648f4b233e06efd4821dd"
format_stamp: "Formatted at 2026-09-26 03:58:40 on dist-test-slave-nhpw"
server_key: "752d55c54a21e00d6c81c77012ff0c1b"
server_key_iv: "ae8141fd35f496927b5bdaf37acb96a3"
server_key_version: "encryptionkey@0"
I20260926 03:58:40.947099 5079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395095474643-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395095474643-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395095474643-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:58:40.951233 5079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:58:40.951521 5079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:58:40.951663 5079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:58:40.951896 5079 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:58:40.952205 5079 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:58:40.952236 5079 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:40.952256 5079 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:58:40.952302 5079 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:40.957755 5079 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36091
I20260926 03:58:40.957818 5206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36091 every 8 connection(s)
I20260926 03:58:40.958166 5079 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
I20260926 03:58:40.961265 5058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:40.958943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48921 (local address 127.0.98.62:45093)
0926 03:58:40.959007 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:40.959010 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:40.959246 (+ 236us) server_negotiation.cc:407] Connection header received
0926 03:58:40.959708 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:40.959710 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:40.959716 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:40.959795 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:40.961057 (+ 1262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:40.961059 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:40.961059 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:58:40.961063 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:40.961077 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:58:40.961081 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:40.961142 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:40.961201 (+ 59us) server_negotiation.cc:299] Negotiation successful
0926 03:58:40.961213 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 03:58:40.961447 5209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:40.958919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45093 (local address 127.0.98.1:48921)
0926 03:58:40.959114 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:40.959152 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:40.959610 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:40.959860 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:40.959864 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:40.959872 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:40.960912 (+ 1040us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:58:40.960913 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:40.960940 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:58:40.960969 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:58:40.961143 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:58:40.961145 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:58:40.961147 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 03:58:40.961198 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 03:58:40.961243 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":107,"threads_started":1}
I20260926 03:58:40.962014 5207 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45093
I20260926 03:58:40.962126 5207 heartbeater.cc:464] Registering TS with master...
I20260926 03:58:40.962273 5207 heartbeater.cc:511] Master 127.0.98.62:45093 requested a full tablet report, sending...
I20260926 03:58:40.962752 5019 ts_manager.cc:194] Registered new tserver with Master: 3ec23d7a3be648f4b233e06efd4821dd (127.0.98.1:36091)
I20260926 03:58:40.965106 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 5079
I20260926 03:58:40.965214 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395095474643-392-0/minicluster-data/ts-0/wal/instance
I20260926 03:58:40.965382 392 external_mini_cluster.cc:1428] Setting key 5f077fef600bca2746abed5a38d52631
I20260926 03:58:40.966239 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395095474643-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395095474643-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395095474643-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45093
--builtin_ntp_servers=127.0.98.20:35287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 03:58:41.045420 5210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:58:41.045632 5210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:58:41.045653 5210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:58:41.046929 5210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:58:41.046969 5210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:58:41.046994 5210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 03:58:41.048319 5210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395095474643-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395095474643-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5210
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395095474643-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:58:41.048544 5210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:58:41.048862 5210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:58:41.054013 5216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:41.054005 5218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:41.054133 5210 server_base.cc:1034] running on GCE node
W20260926 03:58:41.054005 5215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:41.054476 5210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:58:41.054677 5210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:58:41.055814 5210 hybrid_clock.cc:648] HybridClock initialized: now 1790395121055796 us; error 34 us; skew 500 ppm
I20260926 03:58:41.057243 5210 webserver.cc:460] Webserver started at http://127.0.98.2:39893/ using document root <none> and password file <none>
I20260926 03:58:41.057502 5210 fs_manager.cc:359] Metadata directory not provided
I20260926 03:58:41.057551 5210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:58:41.057724 5210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:58:41.058786 5210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395095474643-392-0/minicluster-data/ts-1/data/instance:
uuid: "09835c7bda944500824796e48cebc04a"
format_stamp: "Formatted at 2026-09-26 03:58:41 on dist-test-slave-nhpw"
server_key: "9663893d4d1f3bfa9a3668e08998f2c8"
server_key_iv: "31e3ec8aa8b2b4c125230afb1e8c1756"
server_key_version: "encryptionkey@0"
I20260926 03:58:41.059150 5210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395095474643-392-0/minicluster-data/ts-1/wal/instance:
uuid: "09835c7bda944500824796e48cebc04a"
format_stamp: "Formatted at 2026-09-26 03:58:41 on dist-test-slave-nhpw"
server_key: "9663893d4d1f3bfa9a3668e08998f2c8"
server_key_iv: "31e3ec8aa8b2b4c125230afb1e8c1756"
server_key_version: "encryptionkey@0"
I20260926 03:58:41.060506 5210 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 03:58:41.061367 5224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:41.061575 5210 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 03:58:41.061662 5210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395095474643-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395095474643-392-0/minicluster-data/ts-1/wal
uuid: "09835c7bda944500824796e48cebc04a"
format_stamp: "Formatted at 2026-09-26 03:58:41 on dist-test-slave-nhpw"
server_key: "9663893d4d1f3bfa9a3668e08998f2c8"
server_key_iv: "31e3ec8aa8b2b4c125230afb1e8c1756"
server_key_version: "encryptionkey@0"
I20260926 03:58:41.061717 5210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395095474643-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395095474643-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395095474643-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:58:41.064962 5210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:58:41.065173 5210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:58:41.065270 5210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:58:41.065430 5210 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:58:41.065693 5210 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:58:41.065725 5210 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:41.065747 5210 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:58:41.065758 5210 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:41.071457 5210 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39113
I20260926 03:58:41.071525 5337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39113 every 8 connection(s)
I20260926 03:58:41.071872 5210 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
I20260926 03:58:41.074836 5058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:41.072699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36119 (local address 127.0.98.62:45093)
0926 03:58:41.072770 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:41.072772 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:41.073030 (+ 258us) server_negotiation.cc:407] Connection header received
0926 03:58:41.073488 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:41.073489 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:41.073495 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:41.073547 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:41.074622 (+ 1075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:41.074623 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:41.074624 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:58:41.074628 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:41.074642 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:58:41.074646 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:41.074692 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:41.074771 (+ 79us) server_negotiation.cc:299] Negotiation successful
0926 03:58:41.074783 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 03:58:41.075029 5340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:41.072637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45093 (local address 127.0.98.2:36119)
0926 03:58:41.072869 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:41.072904 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:41.073403 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:41.073600 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:41.073603 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:41.073611 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:41.074479 (+ 868us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:58:41.074479 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:41.074508 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:58:41.074536 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:58:41.074705 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:58:41.074708 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:58:41.074710 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 03:58:41.074767 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 03:58:41.074814 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":109,"threads_started":1}
I20260926 03:58:41.075467 5338 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45093
I20260926 03:58:41.075564 5338 heartbeater.cc:464] Registering TS with master...
I20260926 03:58:41.075728 5338 heartbeater.cc:511] Master 127.0.98.62:45093 requested a full tablet report, sending...
I20260926 03:58:41.076124 5019 ts_manager.cc:194] Registered new tserver with Master: 09835c7bda944500824796e48cebc04a (127.0.98.2:39113)
I20260926 03:58:41.080780 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 5210
I20260926 03:58:41.080862 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395095474643-392-0/minicluster-data/ts-1/wal/instance
I20260926 03:58:41.081029 392 external_mini_cluster.cc:1428] Setting key bc49a317673511d0b01c42caa3b2d8e2
I20260926 03:58:41.081845 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395095474643-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395095474643-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395095474643-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45093
--builtin_ntp_servers=127.0.98.20:35287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 03:58:41.162892 5341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:58:41.163129 5341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:58:41.163167 5341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:58:41.164388 5341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:58:41.164443 5341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:58:41.164482 5341 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 03:58:41.165884 5341 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395095474643-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395095474643-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5341
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395095474643-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:58:41.166138 5341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:58:41.166415 5341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:58:41.171622 5346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:41.171593 5347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:41.171748 5341 server_base.cc:1034] running on GCE node
W20260926 03:58:41.171608 5349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:41.172073 5341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:58:41.172279 5341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:58:41.173494 5341 hybrid_clock.cc:648] HybridClock initialized: now 1790395121173466 us; error 45 us; skew 500 ppm
I20260926 03:58:41.174726 5341 webserver.cc:460] Webserver started at http://127.0.98.3:36033/ using document root <none> and password file <none>
I20260926 03:58:41.174918 5341 fs_manager.cc:359] Metadata directory not provided
I20260926 03:58:41.174959 5341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:58:41.175078 5341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:58:41.176198 5341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395095474643-392-0/minicluster-data/ts-2/data/instance:
uuid: "7696203734fc497b9afb0322ac1edcee"
format_stamp: "Formatted at 2026-09-26 03:58:41 on dist-test-slave-nhpw"
server_key: "00b85eba8d9d748152bccbcbd3b268d1"
server_key_iv: "f9e2d29466f0ac839b37148197616986"
server_key_version: "encryptionkey@0"
I20260926 03:58:41.176499 5341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395095474643-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7696203734fc497b9afb0322ac1edcee"
format_stamp: "Formatted at 2026-09-26 03:58:41 on dist-test-slave-nhpw"
server_key: "00b85eba8d9d748152bccbcbd3b268d1"
server_key_iv: "f9e2d29466f0ac839b37148197616986"
server_key_version: "encryptionkey@0"
I20260926 03:58:41.177835 5341 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 03:58:41.178624 5355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:41.178782 5341 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 03:58:41.178867 5341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395095474643-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395095474643-392-0/minicluster-data/ts-2/wal
uuid: "7696203734fc497b9afb0322ac1edcee"
format_stamp: "Formatted at 2026-09-26 03:58:41 on dist-test-slave-nhpw"
server_key: "00b85eba8d9d748152bccbcbd3b268d1"
server_key_iv: "f9e2d29466f0ac839b37148197616986"
server_key_version: "encryptionkey@0"
I20260926 03:58:41.178915 5341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395095474643-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395095474643-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395095474643-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:58:41.182116 5341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:58:41.182322 5341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:58:41.182412 5341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:58:41.182555 5341 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:58:41.182780 5341 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:58:41.182808 5341 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:41.182827 5341 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:58:41.182839 5341 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:41.188344 5341 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42979
I20260926 03:58:41.188417 5468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42979 every 8 connection(s)
I20260926 03:58:41.188817 5341 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
I20260926 03:58:41.192049 5058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:41.189572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41125 (local address 127.0.98.62:45093)
0926 03:58:41.189633 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:41.189635 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:41.189891 (+ 256us) server_negotiation.cc:407] Connection header received
0926 03:58:41.190432 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:41.190433 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:41.190440 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:41.190504 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:41.191779 (+ 1275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:41.191781 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:41.191781 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:58:41.191785 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:41.191801 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:58:41.191805 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:41.191874 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:41.191964 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 03:58:41.191979 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 03:58:41.192265 5471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:41.189572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45093 (local address 127.0.98.3:41125)
0926 03:58:41.189773 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:41.189806 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:41.190307 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:41.190583 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:41.190592 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:41.190600 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:41.191592 (+ 992us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:58:41.191593 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:41.191623 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:58:41.191675 (+ 52us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:58:41.191871 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:58:41.191877 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:58:41.191880 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 03:58:41.191975 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 03:58:41.192037 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":94,"threads_started":1}
I20260926 03:58:41.192785 5469 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45093
I20260926 03:58:41.192914 5469 heartbeater.cc:464] Registering TS with master...
I20260926 03:58:41.193063 5469 heartbeater.cc:511] Master 127.0.98.62:45093 requested a full tablet report, sending...
I20260926 03:58:41.193485 5019 ts_manager.cc:194] Registered new tserver with Master: 7696203734fc497b9afb0322ac1edcee (127.0.98.3:42979)
I20260926 03:58:41.196071 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 5341
I20260926 03:58:41.196190 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395095474643-392-0/minicluster-data/ts-2/wal/instance
I20260926 03:58:41.196367 392 external_mini_cluster.cc:1428] Setting key 2a927490a7b75eab7896e1e1f99842fb
I20260926 03:58:41.197178 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:58:41.199540 5058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:41.198642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49248 (local address 127.0.98.62:45093)
0926 03:58:41.198683 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:41.198685 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:41.198709 (+ 24us) server_negotiation.cc:407] Connection header received
0926 03:58:41.198760 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:41.198760 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:41.198765 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:41.198830 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:41.199364 (+ 534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:41.199365 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:41.199365 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:58:41.199369 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:41.199381 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:58:41.199384 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:41.199430 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:41.199484 (+ 54us) server_negotiation.cc:299] Negotiation successful
0926 03:58:41.199494 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 03:58:41.201011 5019 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49248:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:58:41.202494 5019 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:58:41.206403 5479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:41.204520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36091 (local address 127.0.0.1:47456)
0926 03:58:41.205149 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:41.205161 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:41.205206 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:41.205658 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:41.205658 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:41.205661 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:41.206043 (+ 382us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:58:41.206043 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:41.206050 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:58:41.206064 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:58:41.206247 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:58:41.206252 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:58:41.206255 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 03:58:41.206321 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 03:58:41.206335 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"thread_start_us":93,"threads_started":1}
I20260926 03:58:41.206607 5477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:41.204547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39113 (local address 127.0.0.1:50444)
0926 03:58:41.205271 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:41.205280 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:41.205316 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:41.205748 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:41.205750 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:41.205752 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:41.206230 (+ 478us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:58:41.206230 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:41.206236 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:58:41.206246 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:58:41.206464 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:58:41.206465 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:58:41.206466 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 03:58:41.206526 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 03:58:41.206539 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":716,"thread_start_us":68,"threads_started":1}
I20260926 03:58:41.206745 5482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:41.204780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47456 (local address 127.0.98.1:36091)
0926 03:58:41.205101 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:41.205104 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:41.205252 (+ 148us) server_negotiation.cc:407] Connection header received
0926 03:58:41.205567 (+ 315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:41.205574 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:41.205584 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:41.205658 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:41.206129 (+ 471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:41.206134 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:41.206135 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:58:41.206138 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:41.206165 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:58:41.206180 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:41.206253 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:41.206589 (+ 336us) server_negotiation.cc:299] Negotiation successful
0926 03:58:41.206606 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":103,"threads_started":1}
I20260926 03:58:41.206756 5478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:41.204488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42979 (local address 127.0.0.1:36524)
0926 03:58:41.204925 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:41.204962 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:41.205027 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:41.205292 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:41.205298 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:41.205304 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:41.205858 (+ 554us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:58:41.205859 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:41.205876 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:58:41.205896 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:58:41.206430 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:58:41.206431 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:58:41.206432 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 03:58:41.206564 (+ 132us) client_negotiation.cc:240] Negotiation successful
0926 03:58:41.206577 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":62,"threads_started":1}
I20260926 03:58:41.206871 5480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:41.204756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50444 (local address 127.0.98.2:39113)
0926 03:58:41.204928 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:41.204953 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:41.205371 (+ 418us) server_negotiation.cc:407] Connection header received
0926 03:58:41.205400 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:41.205403 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:41.205412 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:41.205478 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:41.206317 (+ 839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:41.206323 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:41.206323 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:58:41.206328 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:41.206359 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:58:41.206385 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:41.206438 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:41.206765 (+ 327us) server_negotiation.cc:299] Negotiation successful
0926 03:58:41.206782 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":82,"threads_started":1}
I20260926 03:58:41.206966 5481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:41.204748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36524 (local address 127.0.98.3:42979)
0926 03:58:41.205016 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:41.205019 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:41.205034 (+ 15us) server_negotiation.cc:407] Connection header received
0926 03:58:41.205114 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:41.205120 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:41.205131 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:41.205217 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:41.206302 (+ 1085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:41.206306 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:41.206306 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:58:41.206310 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:41.206339 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:58:41.206352 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:41.206436 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:41.206635 (+ 199us) server_negotiation.cc:299] Negotiation successful
0926 03:58:41.206653 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":91,"threads_started":1}
I20260926 03:58:41.207417 5141 tablet_service.cc:1467] Processing CreateTablet for tablet e328e52ef73d488e9843fca81d543925 (DEFAULT_TABLE table=test-table [id=39db6fd943234c40b5f610811a98fafd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:58:41.207561 5272 tablet_service.cc:1467] Processing CreateTablet for tablet e328e52ef73d488e9843fca81d543925 (DEFAULT_TABLE table=test-table [id=39db6fd943234c40b5f610811a98fafd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:58:41.207810 5141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e328e52ef73d488e9843fca81d543925. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:58:41.207917 5272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e328e52ef73d488e9843fca81d543925. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:58:41.210553 5483 tablet_bootstrap.cc:492] T e328e52ef73d488e9843fca81d543925 P 09835c7bda944500824796e48cebc04a: Bootstrap starting.
I20260926 03:58:41.210603 5484 tablet_bootstrap.cc:492] T e328e52ef73d488e9843fca81d543925 P 3ec23d7a3be648f4b233e06efd4821dd: Bootstrap starting.
I20260926 03:58:41.211035 5403 tablet_service.cc:1467] Processing CreateTablet for tablet e328e52ef73d488e9843fca81d543925 (DEFAULT_TABLE table=test-table [id=39db6fd943234c40b5f610811a98fafd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:58:41.211165 5484 tablet_bootstrap.cc:654] T e328e52ef73d488e9843fca81d543925 P 3ec23d7a3be648f4b233e06efd4821dd: Neither blocks nor log segments found. Creating new log.
I20260926 03:58:41.211265 5483 tablet_bootstrap.cc:654] T e328e52ef73d488e9843fca81d543925 P 09835c7bda944500824796e48cebc04a: Neither blocks nor log segments found. Creating new log.
I20260926 03:58:41.211350 5403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e328e52ef73d488e9843fca81d543925. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:58:41.211508 5484 log.cc:824] T e328e52ef73d488e9843fca81d543925 P 3ec23d7a3be648f4b233e06efd4821dd: Log is configured to *not* fsync() on all Append() calls
I20260926 03:58:41.211580 5483 log.cc:824] T e328e52ef73d488e9843fca81d543925 P 09835c7bda944500824796e48cebc04a: Log is configured to *not* fsync() on all Append() calls
I20260926 03:58:41.212347 5483 tablet_bootstrap.cc:492] T e328e52ef73d488e9843fca81d543925 P 09835c7bda944500824796e48cebc04a: No bootstrap required, opened a new log
I20260926 03:58:41.212419 5483 ts_tablet_manager.cc:1397] T e328e52ef73d488e9843fca81d543925 P 09835c7bda944500824796e48cebc04a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 03:58:41.213374 5484 tablet_bootstrap.cc:492] T e328e52ef73d488e9843fca81d543925 P 3ec23d7a3be648f4b233e06efd4821dd: No bootstrap required, opened a new log
I20260926 03:58:41.213462 5484 ts_tablet_manager.cc:1397] T e328e52ef73d488e9843fca81d543925 P 3ec23d7a3be648f4b233e06efd4821dd: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 03:58:41.214694 5487 tablet_bootstrap.cc:492] T e328e52ef73d488e9843fca81d543925 P 7696203734fc497b9afb0322ac1edcee: Bootstrap starting.
I20260926 03:58:41.215025 5483 raft_consensus.cc:348] T e328e52ef73d488e9843fca81d543925 P 09835c7bda944500824796e48cebc04a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ec23d7a3be648f4b233e06efd4821dd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36091 } } peers { permanent_uuid: "7696203734fc497b9afb0322ac1edcee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42979 } } peers { permanent_uuid: "09835c7bda944500824796e48cebc04a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39113 } }
I20260926 03:58:41.215180 5483 raft_consensus.cc:374] T e328e52ef73d488e9843fca81d543925 P 09835c7bda944500824796e48cebc04a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:58:41.215221 5483 raft_consensus.cc:729] T e328e52ef73d488e9843fca81d543925 P 09835c7bda944500824796e48cebc04a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09835c7bda944500824796e48cebc04a, State: Initialized, Role: FOLLOWER
I20260926 03:58:41.215328 5483 consensus_queue.cc:260] T e328e52ef73d488e9843fca81d543925 P 09835c7bda944500824796e48cebc04a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ec23d7a3be648f4b233e06efd4821dd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36091 } } peers { permanent_uuid: "7696203734fc497b9afb0322ac1edcee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42979 } } peers { permanent_uuid: "09835c7bda944500824796e48cebc04a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39113 } }
I20260926 03:58:41.215546 5487 tablet_bootstrap.cc:654] T e328e52ef73d488e9843fca81d543925 P 7696203734fc497b9afb0322ac1edcee: Neither blocks nor log segments found. Creating new log.
I20260926 03:58:41.215607 5483 ts_tablet_manager.cc:1428] T e328e52ef73d488e9843fca81d543925 P 09835c7bda944500824796e48cebc04a: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260926 03:58:41.215782 5338 heartbeater.cc:503] Master 127.0.98.62:45093 was elected leader, sending a full tablet report...
I20260926 03:58:41.215854 5487 log.cc:824] T e328e52ef73d488e9843fca81d543925 P 7696203734fc497b9afb0322ac1edcee: Log is configured to *not* fsync() on all Append() calls
I20260926 03:58:41.215826 5484 raft_consensus.cc:348] T e328e52ef73d488e9843fca81d543925 P 3ec23d7a3be648f4b233e06efd4821dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ec23d7a3be648f4b233e06efd4821dd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36091 } } peers { permanent_uuid: "7696203734fc497b9afb0322ac1edcee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42979 } } peers { permanent_uuid: "09835c7bda944500824796e48cebc04a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39113 } }
I20260926 03:58:41.215996 5484 raft_consensus.cc:374] T e328e52ef73d488e9843fca81d543925 P 3ec23d7a3be648f4b233e06efd4821dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:58:41.216046 5484 raft_consensus.cc:729] T e328e52ef73d488e9843fca81d543925 P 3ec23d7a3be648f4b233e06efd4821dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ec23d7a3be648f4b233e06efd4821dd, State: Initialized, Role: FOLLOWER
I20260926 03:58:41.216213 5484 consensus_queue.cc:260] T e328e52ef73d488e9843fca81d543925 P 3ec23d7a3be648f4b233e06efd4821dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ec23d7a3be648f4b233e06efd4821dd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36091 } } peers { permanent_uuid: "7696203734fc497b9afb0322ac1edcee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42979 } } peers { permanent_uuid: "09835c7bda944500824796e48cebc04a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39113 } }
I20260926 03:58:41.216472 5484 ts_tablet_manager.cc:1428] T e328e52ef73d488e9843fca81d543925 P 3ec23d7a3be648f4b233e06efd4821dd: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 03:58:41.216487 5207 heartbeater.cc:503] Master 127.0.98.62:45093 was elected leader, sending a full tablet report...
I20260926 03:58:41.216876 5487 tablet_bootstrap.cc:492] T e328e52ef73d488e9843fca81d543925 P 7696203734fc497b9afb0322ac1edcee: No bootstrap required, opened a new log
I20260926 03:58:41.216944 5487 ts_tablet_manager.cc:1397] T e328e52ef73d488e9843fca81d543925 P 7696203734fc497b9afb0322ac1edcee: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 03:58:41.218935 5487 raft_consensus.cc:348] T e328e52ef73d488e9843fca81d543925 P 7696203734fc497b9afb0322ac1edcee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ec23d7a3be648f4b233e06efd4821dd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36091 } } peers { permanent_uuid: "7696203734fc497b9afb0322ac1edcee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42979 } } peers { permanent_uuid: "09835c7bda944500824796e48cebc04a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39113 } }
I20260926 03:58:41.219077 5487 raft_consensus.cc:374] T e328e52ef73d488e9843fca81d543925 P 7696203734fc497b9afb0322ac1edcee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:58:41.219120 5487 raft_consensus.cc:729] T e328e52ef73d488e9843fca81d543925 P 7696203734fc497b9afb0322ac1edcee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7696203734fc497b9afb0322ac1edcee, State: Initialized, Role: FOLLOWER
I20260926 03:58:41.219224 5487 consensus_queue.cc:260] T e328e52ef73d488e9843fca81d543925 P 7696203734fc497b9afb0322ac1edcee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ec23d7a3be648f4b233e06efd4821dd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36091 } } peers { permanent_uuid: "7696203734fc497b9afb0322ac1edcee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42979 } } peers { permanent_uuid: "09835c7bda944500824796e48cebc04a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39113 } }
I20260926 03:58:41.219475 5487 ts_tablet_manager.cc:1428] T e328e52ef73d488e9843fca81d543925 P 7696203734fc497b9afb0322ac1edcee: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 03:58:41.219547 5469 heartbeater.cc:503] Master 127.0.98.62:45093 was elected leader, sending a full tablet report...
I20260926 03:58:41.237960 5488 raft_consensus.cc:482] T e328e52ef73d488e9843fca81d543925 P 09835c7bda944500824796e48cebc04a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:58:41.238096 5488 raft_consensus.cc:504] T e328e52ef73d488e9843fca81d543925 P 09835c7bda944500824796e48cebc04a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ec23d7a3be648f4b233e06efd4821dd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36091 } } peers { permanent_uuid: "7696203734fc497b9afb0322ac1edcee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42979 } } peers { permanent_uuid: "09835c7bda944500824796e48cebc04a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39113 } }
I20260926 03:58:41.238520 5488 leader_election.cc:288] T e328e52ef73d488e9843fca81d543925 P 09835c7bda944500824796e48cebc04a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3ec23d7a3be648f4b233e06efd4821dd (127.0.98.1:36091), 7696203734fc497b9afb0322ac1edcee (127.0.98.3:42979)
I20260926 03:58:41.239754 5482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:41.238734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52127 (local address 127.0.98.1:36091)
0926 03:58:41.238782 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:41.238784 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:41.238795 (+ 11us) server_negotiation.cc:407] Connection header received
0926 03:58:41.238865 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:41.238866 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:41.238870 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:41.238920 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:41.239496 (+ 576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:41.239497 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:41.239498 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:58:41.239502 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:41.239519 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:58:41.239523 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:41.239605 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:41.239655 (+ 50us) server_negotiation.cc:299] Negotiation successful
0926 03:58:41.239669 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 03:58:41.239765 5340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:41.238674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36091 (local address 127.0.98.2:52127)
0926 03:58:41.238730 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:41.238743 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:41.238783 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:41.238948 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:41.238949 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:41.238951 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:41.239373 (+ 422us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:58:41.239373 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:41.239380 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:58:41.239392 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:58:41.239601 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:58:41.239602 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:58:41.239603 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 03:58:41.239645 (+ 42us) client_negotiation.cc:240] Negotiation successful
0926 03:58:41.239683 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 03:58:41.239830 5481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:41.238758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34475 (local address 127.0.98.3:42979)
0926 03:58:41.238802 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:41.238804 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:41.238985 (+ 181us) server_negotiation.cc:407] Connection header received
0926 03:58:41.239013 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:41.239014 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:41.239025 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:41.239067 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:41.239608 (+ 541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:41.239609 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:41.239610 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:58:41.239615 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:41.239631 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:58:41.239635 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:41.239691 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:41.239745 (+ 54us) server_negotiation.cc:299] Negotiation successful
0926 03:58:41.239757 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 03:58:41.239882 5492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:41.238690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42979 (local address 127.0.98.2:34475)
0926 03:58:41.238872 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:41.238881 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:41.238957 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:41.239098 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:41.239099 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:41.239102 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:41.239512 (+ 410us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:58:41.239512 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:41.239517 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:58:41.239526 (+ 9us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:58:41.239698 (+ 172us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:58:41.239699 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:58:41.239700 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 03:58:41.239747 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 03:58:41.239757 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":115,"threads_started":1}
I20260926 03:58:41.240118 5161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e328e52ef73d488e9843fca81d543925" candidate_uuid: "09835c7bda944500824796e48cebc04a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ec23d7a3be648f4b233e06efd4821dd" is_pre_election: true
I20260926 03:58:41.240247 5423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e328e52ef73d488e9843fca81d543925" candidate_uuid: "09835c7bda944500824796e48cebc04a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7696203734fc497b9afb0322ac1edcee" is_pre_election: true
I20260926 03:58:41.240293 5161 raft_consensus.cc:2454] T e328e52ef73d488e9843fca81d543925 P 3ec23d7a3be648f4b233e06efd4821dd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09835c7bda944500824796e48cebc04a in term 0.
I20260926 03:58:41.240406 5423 raft_consensus.cc:2454] T e328e52ef73d488e9843fca81d543925 P 7696203734fc497b9afb0322ac1edcee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09835c7bda944500824796e48cebc04a in term 0.
I20260926 03:58:41.240520 5226 leader_election.cc:302] T e328e52ef73d488e9843fca81d543925 P 09835c7bda944500824796e48cebc04a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09835c7bda944500824796e48cebc04a, 3ec23d7a3be648f4b233e06efd4821dd; no voters:
I20260926 03:58:41.240784 5488 raft_consensus.cc:2781] T e328e52ef73d488e9843fca81d543925 P 09835c7bda944500824796e48cebc04a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:58:41.240845 5488 raft_consensus.cc:482] T e328e52ef73d488e9843fca81d543925 P 09835c7bda944500824796e48cebc04a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:58:41.240875 5488 raft_consensus.cc:3037] T e328e52ef73d488e9843fca81d543925 P 09835c7bda944500824796e48cebc04a [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:58:41.241590 5488 raft_consensus.cc:504] T e328e52ef73d488e9843fca81d543925 P 09835c7bda944500824796e48cebc04a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ec23d7a3be648f4b233e06efd4821dd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36091 } } peers { permanent_uuid: "7696203734fc497b9afb0322ac1edcee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42979 } } peers { permanent_uuid: "09835c7bda944500824796e48cebc04a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39113 } }
I20260926 03:58:41.241735 5488 leader_election.cc:288] T e328e52ef73d488e9843fca81d543925 P 09835c7bda944500824796e48cebc04a [CANDIDATE]: Term 1 election: Requested vote from peers 3ec23d7a3be648f4b233e06efd4821dd (127.0.98.1:36091), 7696203734fc497b9afb0322ac1edcee (127.0.98.3:42979)
I20260926 03:58:41.241866 5161 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e328e52ef73d488e9843fca81d543925" candidate_uuid: "09835c7bda944500824796e48cebc04a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ec23d7a3be648f4b233e06efd4821dd"
I20260926 03:58:41.241921 5423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e328e52ef73d488e9843fca81d543925" candidate_uuid: "09835c7bda944500824796e48cebc04a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7696203734fc497b9afb0322ac1edcee"
I20260926 03:58:41.241947 5161 raft_consensus.cc:3037] T e328e52ef73d488e9843fca81d543925 P 3ec23d7a3be648f4b233e06efd4821dd [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:58:41.242002 5423 raft_consensus.cc:3037] T e328e52ef73d488e9843fca81d543925 P 7696203734fc497b9afb0322ac1edcee [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:58:41.242664 5161 raft_consensus.cc:2454] T e328e52ef73d488e9843fca81d543925 P 3ec23d7a3be648f4b233e06efd4821dd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09835c7bda944500824796e48cebc04a in term 1.
I20260926 03:58:41.242740 5423 raft_consensus.cc:2454] T e328e52ef73d488e9843fca81d543925 P 7696203734fc497b9afb0322ac1edcee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09835c7bda944500824796e48cebc04a in term 1.
I20260926 03:58:41.242818 5226 leader_election.cc:302] T e328e52ef73d488e9843fca81d543925 P 09835c7bda944500824796e48cebc04a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09835c7bda944500824796e48cebc04a, 3ec23d7a3be648f4b233e06efd4821dd; no voters:
I20260926 03:58:41.242914 5488 raft_consensus.cc:2781] T e328e52ef73d488e9843fca81d543925 P 09835c7bda944500824796e48cebc04a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:58:41.243076 5488 raft_consensus.cc:686] T e328e52ef73d488e9843fca81d543925 P 09835c7bda944500824796e48cebc04a [term 1 LEADER]: Becoming Leader. State: Replica: 09835c7bda944500824796e48cebc04a, State: Running, Role: LEADER
I20260926 03:58:41.243181 5488 consensus_queue.cc:237] T e328e52ef73d488e9843fca81d543925 P 09835c7bda944500824796e48cebc04a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ec23d7a3be648f4b233e06efd4821dd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36091 } } peers { permanent_uuid: "7696203734fc497b9afb0322ac1edcee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42979 } } peers { permanent_uuid: "09835c7bda944500824796e48cebc04a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39113 } }
I20260926 03:58:41.243952 5018 catalog_manager.cc:5495] T e328e52ef73d488e9843fca81d543925 P 09835c7bda944500824796e48cebc04a reported cstate change: term changed from 0 to 1, leader changed from <none> to 09835c7bda944500824796e48cebc04a (127.0.98.2). New cstate: current_term: 1 leader_uuid: "09835c7bda944500824796e48cebc04a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3ec23d7a3be648f4b233e06efd4821dd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36091 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7696203734fc497b9afb0322ac1edcee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09835c7bda944500824796e48cebc04a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39113 } health_report { overall_health: HEALTHY } } }
I20260926 03:58:41.253747 5480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:41.252686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50446 (local address 127.0.98.2:39113)
0926 03:58:41.252777 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:41.252779 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:41.252786 (+ 7us) server_negotiation.cc:407] Connection header received
0926 03:58:41.252817 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:41.252818 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:41.252828 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:41.252881 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:41.253475 (+ 594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:41.253476 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:41.253476 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:58:41.253480 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:41.253495 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:58:41.253499 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:41.253558 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:41.253613 (+ 55us) server_negotiation.cc:299] Negotiation successful
0926 03:58:41.253625 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 03:58:41.255561 5161 raft_consensus.cc:1261] T e328e52ef73d488e9843fca81d543925 P 3ec23d7a3be648f4b233e06efd4821dd [term 1 FOLLOWER]: Refusing update from remote peer 09835c7bda944500824796e48cebc04a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:58:41.255589 5423 raft_consensus.cc:1261] T e328e52ef73d488e9843fca81d543925 P 7696203734fc497b9afb0322ac1edcee [term 1 FOLLOWER]: Refusing update from remote peer 09835c7bda944500824796e48cebc04a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:58:41.255923 5488 consensus_queue.cc:1035] T e328e52ef73d488e9843fca81d543925 P 09835c7bda944500824796e48cebc04a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ec23d7a3be648f4b233e06efd4821dd" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36091 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:58:41.256070 5493 consensus_queue.cc:1035] T e328e52ef73d488e9843fca81d543925 P 09835c7bda944500824796e48cebc04a [LEADER]: Connected to new peer: Peer: permanent_uuid: "7696203734fc497b9afb0322ac1edcee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:58:41.258705 5498 mvcc.cc:205] Tried to move back new op lower bound from 7333458416659992576 to 7333458416612679680. Current Snapshot: MvccSnapshot[applied={T|T < 7333458416659992576}]
I20260926 03:58:41.259578 5497 mvcc.cc:205] Tried to move back new op lower bound from 7333458416659992576 to 7333458416612679680. Current Snapshot: MvccSnapshot[applied={T|T < 7333458416659992576}]
I20260926 03:58:41.260752 5019 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49248:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:58:41.260846 5019 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49248:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:58:41.261535 5019 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c68f8cac93a84d68bdd27e505c5ec226: Sending DeleteTablet for 3 replicas of tablet e328e52ef73d488e9843fca81d543925
I20260926 03:58:41.261854 5403 tablet_service.cc:1514] Processing DeleteTablet for tablet e328e52ef73d488e9843fca81d543925 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:58:41 UTC) from {username='slave'} at 127.0.0.1:36524
I20260926 03:58:41.261855 5272 tablet_service.cc:1514] Processing DeleteTablet for tablet e328e52ef73d488e9843fca81d543925 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:58:41 UTC) from {username='slave'} at 127.0.0.1:50444
I20260926 03:58:41.261879 5141 tablet_service.cc:1514] Processing DeleteTablet for tablet e328e52ef73d488e9843fca81d543925 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:58:41 UTC) from {username='slave'} at 127.0.0.1:47456
I20260926 03:58:41.262097 5506 tablet_replica.cc:330] T e328e52ef73d488e9843fca81d543925 P 09835c7bda944500824796e48cebc04a: stopping tablet replica
I20260926 03:58:41.262182 5507 tablet_replica.cc:330] T e328e52ef73d488e9843fca81d543925 P 7696203734fc497b9afb0322ac1edcee: stopping tablet replica
I20260926 03:58:41.262226 5505 tablet_replica.cc:330] T e328e52ef73d488e9843fca81d543925 P 3ec23d7a3be648f4b233e06efd4821dd: stopping tablet replica
I20260926 03:58:41.262319 5505 raft_consensus.cc:2229] T e328e52ef73d488e9843fca81d543925 P 3ec23d7a3be648f4b233e06efd4821dd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:58:41.262370 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 5079
I20260926 03:58:41.262346 5507 raft_consensus.cc:2229] T e328e52ef73d488e9843fca81d543925 P 7696203734fc497b9afb0322ac1edcee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:58:41.262557 5507 raft_consensus.cc:2258] T e328e52ef73d488e9843fca81d543925 P 7696203734fc497b9afb0322ac1edcee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:58:41.263056 5507 ts_tablet_manager.cc:1905] T e328e52ef73d488e9843fca81d543925 P 7696203734fc497b9afb0322ac1edcee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:58:41.263734 5506 raft_consensus.cc:2229] T e328e52ef73d488e9843fca81d543925 P 09835c7bda944500824796e48cebc04a [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:58:41.263850 5506 raft_consensus.cc:2258] T e328e52ef73d488e9843fca81d543925 P 09835c7bda944500824796e48cebc04a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:58:41.264317 5506 ts_tablet_manager.cc:1905] T e328e52ef73d488e9843fca81d543925 P 09835c7bda944500824796e48cebc04a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:58:41.265039 5507 ts_tablet_manager.cc:1918] T e328e52ef73d488e9843fca81d543925 P 7696203734fc497b9afb0322ac1edcee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:58:41.265164 5507 log.cc:1196] T e328e52ef73d488e9843fca81d543925 P 7696203734fc497b9afb0322ac1edcee: Deleting WAL directory at /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395095474643-392-0/minicluster-data/ts-2/wal/wals/e328e52ef73d488e9843fca81d543925
I20260926 03:58:41.265509 5507 ts_tablet_manager.cc:1939] T e328e52ef73d488e9843fca81d543925 P 7696203734fc497b9afb0322ac1edcee: Deleting consensus metadata
I20260926 03:58:41.265987 5005 catalog_manager.cc:4841] TS 7696203734fc497b9afb0322ac1edcee (127.0.98.3:42979): tablet e328e52ef73d488e9843fca81d543925 (table test-table [id=39db6fd943234c40b5f610811a98fafd]) successfully deleted
I20260926 03:58:41.266479 5506 ts_tablet_manager.cc:1918] T e328e52ef73d488e9843fca81d543925 P 09835c7bda944500824796e48cebc04a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:58:41.266556 5506 log.cc:1196] T e328e52ef73d488e9843fca81d543925 P 09835c7bda944500824796e48cebc04a: Deleting WAL directory at /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790395095474643-392-0/minicluster-data/ts-1/wal/wals/e328e52ef73d488e9843fca81d543925
I20260926 03:58:41.266829 5506 ts_tablet_manager.cc:1939] T e328e52ef73d488e9843fca81d543925 P 09835c7bda944500824796e48cebc04a: Deleting consensus metadata
I20260926 03:58:41.267233 5006 catalog_manager.cc:4841] TS 09835c7bda944500824796e48cebc04a (127.0.98.2:39113): tablet e328e52ef73d488e9843fca81d543925 (table test-table [id=39db6fd943234c40b5f610811a98fafd]) successfully deleted
W20260926 03:58:41.269408 5004 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36091 (error 108)
I20260926 03:58:41.269608 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 5210
I20260926 03:58:41.271865 5478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:41.271710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36091 (local address 127.0.0.1:47468)
0926 03:58:41.271776 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:41.271826 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 03:58:41.271996 5004 catalog_manager.cc:4584] TS 3ec23d7a3be648f4b233e06efd4821dd (127.0.98.1:36091): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e328e52ef73d488e9843fca81d543925: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36091: connect: Connection refused (error 111)
I20260926 03:58:41.275074 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 5341
I20260926 03:58:41.279400 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 4988
2026-09-26T03:58:41Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (610 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260926 03:58:41.298964 392 mini_oidc.cc:152] Starting JWKS server
I20260926 03:58:41.299983 392 webserver.cc:460] Webserver started at https://127.0.0.1:43057/ using document root <none> and password file <none>
I20260926 03:58:41.300107 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 03:58:41.300280 392 webserver.cc:460] Webserver started at http://127.0.0.1:40875/ using document root <none> and password file <none>
2026-09-26T03:58:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:58:41Z Disabled control of system clock
I20260926 03:58:41.307958 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790395095474643-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790395095474643-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790395095474643-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46813
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38303
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46813
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43057/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790395095474643-392-0/testchainca.cert with env {}
W20260926 03:58:41.386188 5514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:58:41.386375 5514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:58:41.386397 5514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:58:41.387481 5514 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:58:41.387514 5514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:58:41.387526 5514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:58:41.387539 5514 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:58:41.387552 5514 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 03:58:41.387563 5514 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 03:58:41.387574 5514 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43057/jwks.json
I20260926 03:58:41.388994 5514 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38303
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790395095474643-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790395095474643-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46813
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43057/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790395095474643-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5514
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790395095474643-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:58:41.389174 5514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:58:41.389325 5514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:58:41.393896 5520 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:41.393896 5522 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:41.393920 5519 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:41.394241 5514 server_base.cc:1034] running on GCE node
I20260926 03:58:41.394474 5514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:58:41.394721 5514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:58:41.395913 5514 hybrid_clock.cc:648] HybridClock initialized: now 1790395121395887 us; error 41 us; skew 500 ppm
I20260926 03:58:41.397306 5514 webserver.cc:460] Webserver started at http://127.0.98.62:40113/ using document root <none> and password file <none>
I20260926 03:58:41.397548 5514 fs_manager.cc:359] Metadata directory not provided
I20260926 03:58:41.397624 5514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:58:41.397778 5514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:58:41.398939 5514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790395095474643-392-0/minicluster-data/master-0/data/instance:
uuid: "00d281b3775143b294ede013321d5ea8"
format_stamp: "Formatted at 2026-09-26 03:58:41 on dist-test-slave-nhpw"
server_key: "6f078c4102b93e863af08221ab95beec"
server_key_iv: "bdbe4aa3c8d8b266aa74ceea318eb339"
server_key_version: "encryptionkey@0"
I20260926 03:58:41.399302 5514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790395095474643-392-0/minicluster-data/master-0/wal/instance:
uuid: "00d281b3775143b294ede013321d5ea8"
format_stamp: "Formatted at 2026-09-26 03:58:41 on dist-test-slave-nhpw"
server_key: "6f078c4102b93e863af08221ab95beec"
server_key_iv: "bdbe4aa3c8d8b266aa74ceea318eb339"
server_key_version: "encryptionkey@0"
I20260926 03:58:41.400585 5514 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 03:58:41.401379 5528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:41.401614 5514 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 03:58:41.401705 5514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790395095474643-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790395095474643-392-0/minicluster-data/master-0/wal
uuid: "00d281b3775143b294ede013321d5ea8"
format_stamp: "Formatted at 2026-09-26 03:58:41 on dist-test-slave-nhpw"
server_key: "6f078c4102b93e863af08221ab95beec"
server_key_iv: "bdbe4aa3c8d8b266aa74ceea318eb339"
server_key_version: "encryptionkey@0"
I20260926 03:58:41.401757 5514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790395095474643-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790395095474643-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790395095474643-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:58:41.410566 5514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:58:41.410806 5514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:58:41.410903 5514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:58:41.414788 5514 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46813
I20260926 03:58:41.414817 5580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46813 every 8 connection(s)
I20260926 03:58:41.415143 5514 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
I20260926 03:58:41.416280 5581 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:58:41.418540 5581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00d281b3775143b294ede013321d5ea8: Bootstrap starting.
I20260926 03:58:41.419082 5581 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00d281b3775143b294ede013321d5ea8: Neither blocks nor log segments found. Creating new log.
I20260926 03:58:41.419317 5581 log.cc:824] T 00000000000000000000000000000000 P 00d281b3775143b294ede013321d5ea8: Log is configured to *not* fsync() on all Append() calls
I20260926 03:58:41.419915 5581 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00d281b3775143b294ede013321d5ea8: No bootstrap required, opened a new log
I20260926 03:58:41.421494 5581 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00d281b3775143b294ede013321d5ea8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00d281b3775143b294ede013321d5ea8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46813 } }
I20260926 03:58:41.421625 5581 raft_consensus.cc:374] T 00000000000000000000000000000000 P 00d281b3775143b294ede013321d5ea8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:58:41.421648 5581 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00d281b3775143b294ede013321d5ea8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00d281b3775143b294ede013321d5ea8, State: Initialized, Role: FOLLOWER
I20260926 03:58:41.421779 5581 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00d281b3775143b294ede013321d5ea8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00d281b3775143b294ede013321d5ea8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46813 } }
I20260926 03:58:41.421885 5581 raft_consensus.cc:388] T 00000000000000000000000000000000 P 00d281b3775143b294ede013321d5ea8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:58:41.421926 5581 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00d281b3775143b294ede013321d5ea8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:58:41.421978 5581 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00d281b3775143b294ede013321d5ea8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:58:41.421981 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 5514
I20260926 03:58:41.422111 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790395095474643-392-0/minicluster-data/master-0/wal/instance
I20260926 03:58:41.422299 392 external_mini_cluster.cc:1428] Setting key 452da66b289314ac10daa80b81bf94c6
I20260926 03:58:41.422657 5581 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00d281b3775143b294ede013321d5ea8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00d281b3775143b294ede013321d5ea8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46813 } }
I20260926 03:58:41.422772 5581 leader_election.cc:302] T 00000000000000000000000000000000 P 00d281b3775143b294ede013321d5ea8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00d281b3775143b294ede013321d5ea8; no voters:
I20260926 03:58:41.422927 5581 leader_election.cc:288] T 00000000000000000000000000000000 P 00d281b3775143b294ede013321d5ea8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:58:41.423007 5586 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00d281b3775143b294ede013321d5ea8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:58:41.423154 5586 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00d281b3775143b294ede013321d5ea8 [term 1 LEADER]: Becoming Leader. State: Replica: 00d281b3775143b294ede013321d5ea8, State: Running, Role: LEADER
I20260926 03:58:41.423199 5581 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00d281b3775143b294ede013321d5ea8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:58:41.423290 5586 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00d281b3775143b294ede013321d5ea8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00d281b3775143b294ede013321d5ea8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46813 } }
I20260926 03:58:41.423730 5588 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00d281b3775143b294ede013321d5ea8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00d281b3775143b294ede013321d5ea8. Latest consensus state: current_term: 1 leader_uuid: "00d281b3775143b294ede013321d5ea8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00d281b3775143b294ede013321d5ea8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46813 } } }
I20260926 03:58:41.423801 5587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00d281b3775143b294ede013321d5ea8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "00d281b3775143b294ede013321d5ea8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00d281b3775143b294ede013321d5ea8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46813 } } }
I20260926 03:58:41.423825 5588 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00d281b3775143b294ede013321d5ea8 [sys.catalog]: This master's current role is: LEADER
I20260926 03:58:41.423869 5587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00d281b3775143b294ede013321d5ea8 [sys.catalog]: This master's current role is: LEADER
I20260926 03:58:41.424046 5590 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:58:41.424607 5590 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:58:41.426252 5590 catalog_manager.cc:1347] Generated new cluster ID: c0b895235d5d4be8a57d7f259059ab5c
I20260926 03:58:41.426303 5590 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:58:41.428485 5585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:41.422777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50318 (local address 127.0.98.62:46813)
0926 03:58:41.423007 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:41.423015 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:41.423043 (+ 28us) server_negotiation.cc:407] Connection header received
0926 03:58:41.423166 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:41.423172 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:41.423240 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:41.423621 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:41.423789 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:41.425118 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:41.426231 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:41.426820 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:41.426932 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:41.427786 (+ 854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:41.427790 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:41.427790 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:58:41.427979 (+ 189us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:41.428029 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:58:41.428043 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:41.428116 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:41.428217 (+ 101us) server_negotiation.cc:299] Negotiation successful
0926 03:58:41.428290 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":99,"threads_started":1}
I20260926 03:58:41.429345 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:58:41.462497 5590 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:58:41.463047 5590 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:58:41.469159 5590 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 00d281b3775143b294ede013321d5ea8: Generated new TSK 0
I20260926 03:58:41.469353 5590 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 03:58:41.492003 5611 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260926 03:58:41.492106 5611 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 03:58:41.492339 5585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:41.477296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50320 (local address 127.0.98.62:46813)
0926 03:58:41.477345 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:41.477347 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:41.477492 (+ 145us) server_negotiation.cc:407] Connection header received
0926 03:58:41.477593 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:41.477593 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:41.477604 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:41.477663 (+ 59us) server_negotiation.cc:226] Negotiated authn=JWT
0926 03:58:41.477830 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:41.478299 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:41.479473 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:41.479648 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:41.479701 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:41.479977 (+ 276us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 03:58:41.492127 (+ 12150us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43057/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 03:58:41.492274 (+ 147us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50320: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43057/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":25}
W20260926 03:58:41.492398 5585 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50320: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43057/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260926 03:58:41.492707 5610 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:58:41.477191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46813 (local address 127.0.0.1:50320)
0926 03:58:41.477411 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:41.477424 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:41.477501 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:41.477695 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:41.477696 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:41.477698 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0926 03:58:41.477762 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:41.477764 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:41.478398 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:41.478399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:41.479370 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:41.479373 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:41.479731 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:41.479738 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:41.479829 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:41.479855 (+ 26us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 03:58:41.492277 (+ 12422us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43057/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 03:58:41.492373 (+ 96us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:46813: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43057/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":104,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:46813: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43057/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260926 03:58:41.493147 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 5514
2026-09-26T03:58:41Z chronyd exiting
I20260926 03:58:41.529381 392 test_util.cc:175] -----------------------------------------------
I20260926 03:58:41.529467 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790395095474643-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (231 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260926 03:58:41.530265 392 mini_oidc.cc:152] Starting JWKS server
I20260926 03:58:41.530956 392 webserver.cc:460] Webserver started at https://127.0.0.1:43841/ using document root <none> and password file <none>
I20260926 03:58:41.530995 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 03:58:41.531193 392 webserver.cc:460] Webserver started at http://127.0.0.1:39597/ using document root <none> and password file <none>
2026-09-26T03:58:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:58:41Z Disabled control of system clock
I20260926 03:58:41.539100 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790395095474643-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790395095474643-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790395095474643-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39683
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39683
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43841/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790395095474643-392-0/testchainca.cert with env {}
W20260926 03:58:41.618063 5617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:58:41.618316 5617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:58:41.618356 5617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:58:41.619545 5617 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:58:41.619620 5617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:58:41.619642 5617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:58:41.619664 5617 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:58:41.619694 5617 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 03:58:41.619719 5617 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 03:58:41.619748 5617 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43841/jwks.json
I20260926 03:58:41.621309 5617 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790395095474643-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790395095474643-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39683
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43841/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790395095474643-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790395095474643-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:58:41.621604 5617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:58:41.621824 5617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:58:41.626690 5625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:41.626777 5623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:41.626703 5622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:41.626855 5617 server_base.cc:1034] running on GCE node
I20260926 03:58:41.627127 5617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:58:41.627405 5617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:58:41.628582 5617 hybrid_clock.cc:648] HybridClock initialized: now 1790395121628570 us; error 49 us; skew 500 ppm
I20260926 03:58:41.629853 5617 webserver.cc:460] Webserver started at http://127.0.98.62:46115/ using document root <none> and password file <none>
I20260926 03:58:41.630112 5617 fs_manager.cc:359] Metadata directory not provided
I20260926 03:58:41.630182 5617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:58:41.630326 5617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:58:41.631381 5617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790395095474643-392-0/minicluster-data/master-0/data/instance:
uuid: "3f26942065bc498b85cfbc6d218aa4e0"
format_stamp: "Formatted at 2026-09-26 03:58:41 on dist-test-slave-nhpw"
server_key: "2e8c35767f168a4d4c1130744ae7ad6c"
server_key_iv: "6bccf4ab975e20b9dfbbafd2ee98b393"
server_key_version: "encryptionkey@0"
I20260926 03:58:41.631742 5617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790395095474643-392-0/minicluster-data/master-0/wal/instance:
uuid: "3f26942065bc498b85cfbc6d218aa4e0"
format_stamp: "Formatted at 2026-09-26 03:58:41 on dist-test-slave-nhpw"
server_key: "2e8c35767f168a4d4c1130744ae7ad6c"
server_key_iv: "6bccf4ab975e20b9dfbbafd2ee98b393"
server_key_version: "encryptionkey@0"
I20260926 03:58:41.633299 5617 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 03:58:41.634047 5631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:41.634330 5617 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 03:58:41.634449 5617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790395095474643-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790395095474643-392-0/minicluster-data/master-0/wal
uuid: "3f26942065bc498b85cfbc6d218aa4e0"
format_stamp: "Formatted at 2026-09-26 03:58:41 on dist-test-slave-nhpw"
server_key: "2e8c35767f168a4d4c1130744ae7ad6c"
server_key_iv: "6bccf4ab975e20b9dfbbafd2ee98b393"
server_key_version: "encryptionkey@0"
I20260926 03:58:41.634522 5617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790395095474643-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790395095474643-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790395095474643-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:58:41.641405 5617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:58:41.641680 5617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:58:41.641835 5617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:58:41.645783 5617 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39683
I20260926 03:58:41.645829 5683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39683 every 8 connection(s)
I20260926 03:58:41.646184 5617 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
I20260926 03:58:41.647195 5684 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:58:41.649492 5684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f26942065bc498b85cfbc6d218aa4e0: Bootstrap starting.
I20260926 03:58:41.650106 5684 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f26942065bc498b85cfbc6d218aa4e0: Neither blocks nor log segments found. Creating new log.
I20260926 03:58:41.650461 5684 log.cc:824] T 00000000000000000000000000000000 P 3f26942065bc498b85cfbc6d218aa4e0: Log is configured to *not* fsync() on all Append() calls
I20260926 03:58:41.651158 5684 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f26942065bc498b85cfbc6d218aa4e0: No bootstrap required, opened a new log
I20260926 03:58:41.653060 5684 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f26942065bc498b85cfbc6d218aa4e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f26942065bc498b85cfbc6d218aa4e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39683 } }
I20260926 03:58:41.653200 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 5617
I20260926 03:58:41.653236 5684 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f26942065bc498b85cfbc6d218aa4e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:58:41.653290 5684 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f26942065bc498b85cfbc6d218aa4e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f26942065bc498b85cfbc6d218aa4e0, State: Initialized, Role: FOLLOWER
I20260926 03:58:41.653316 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790395095474643-392-0/minicluster-data/master-0/wal/instance
I20260926 03:58:41.653461 5684 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f26942065bc498b85cfbc6d218aa4e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f26942065bc498b85cfbc6d218aa4e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39683 } }
I20260926 03:58:41.653527 392 external_mini_cluster.cc:1428] Setting key 04a61f5c553ca067663b1a5e60cd8746
I20260926 03:58:41.653548 5684 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f26942065bc498b85cfbc6d218aa4e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:58:41.653590 5684 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f26942065bc498b85cfbc6d218aa4e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:58:41.653642 5684 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f26942065bc498b85cfbc6d218aa4e0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:58:41.654372 5684 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f26942065bc498b85cfbc6d218aa4e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f26942065bc498b85cfbc6d218aa4e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39683 } }
I20260926 03:58:41.654501 5684 leader_election.cc:302] T 00000000000000000000000000000000 P 3f26942065bc498b85cfbc6d218aa4e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f26942065bc498b85cfbc6d218aa4e0; no voters:
I20260926 03:58:41.654675 5684 leader_election.cc:288] T 00000000000000000000000000000000 P 3f26942065bc498b85cfbc6d218aa4e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:58:41.654757 5689 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f26942065bc498b85cfbc6d218aa4e0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:58:41.654903 5689 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f26942065bc498b85cfbc6d218aa4e0 [term 1 LEADER]: Becoming Leader. State: Replica: 3f26942065bc498b85cfbc6d218aa4e0, State: Running, Role: LEADER
I20260926 03:58:41.654955 5684 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f26942065bc498b85cfbc6d218aa4e0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:58:41.655048 5689 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f26942065bc498b85cfbc6d218aa4e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f26942065bc498b85cfbc6d218aa4e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39683 } }
I20260926 03:58:41.655529 5690 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f26942065bc498b85cfbc6d218aa4e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f26942065bc498b85cfbc6d218aa4e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f26942065bc498b85cfbc6d218aa4e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39683 } } }
I20260926 03:58:41.655623 5690 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f26942065bc498b85cfbc6d218aa4e0 [sys.catalog]: This master's current role is: LEADER
I20260926 03:58:41.655826 5691 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f26942065bc498b85cfbc6d218aa4e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f26942065bc498b85cfbc6d218aa4e0. Latest consensus state: current_term: 1 leader_uuid: "3f26942065bc498b85cfbc6d218aa4e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f26942065bc498b85cfbc6d218aa4e0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39683 } } }
I20260926 03:58:41.655956 5691 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f26942065bc498b85cfbc6d218aa4e0 [sys.catalog]: This master's current role is: LEADER
I20260926 03:58:41.656153 5697 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:58:41.656605 5697 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:58:41.658282 5697 catalog_manager.cc:1347] Generated new cluster ID: d041297f4834461997ad85884be71a08
I20260926 03:58:41.658341 5697 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:58:41.659870 5688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:41.654012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36234 (local address 127.0.98.62:39683)
0926 03:58:41.654214 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:41.654221 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:41.654244 (+ 23us) server_negotiation.cc:407] Connection header received
0926 03:58:41.654330 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:41.654336 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:41.654392 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:41.654525 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:41.654740 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:41.656478 (+ 1738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:41.657650 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:41.658252 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:41.658378 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:41.659104 (+ 726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:41.659108 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:41.659108 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:58:41.659298 (+ 190us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:41.659336 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:58:41.659348 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:41.659412 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:41.659515 (+ 103us) server_negotiation.cc:299] Negotiation successful
0926 03:58:41.659669 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":76,"threads_started":1}
I20260926 03:58:41.660799 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:58:41.686345 5697 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:58:41.687115 5697 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:58:41.694197 5697 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f26942065bc498b85cfbc6d218aa4e0: Generated new TSK 0
I20260926 03:58:41.694435 5697 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 03:58:41.732110 5714 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260926 03:58:41.732214 5714 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 03:58:41.732429 5688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:41.709238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36250 (local address 127.0.98.62:39683)
0926 03:58:41.709311 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:41.709313 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:41.709430 (+ 117us) server_negotiation.cc:407] Connection header received
0926 03:58:41.709528 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:41.709529 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:41.709548 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:41.709645 (+ 97us) server_negotiation.cc:226] Negotiated authn=JWT
0926 03:58:41.709805 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:41.710314 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:41.711158 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:41.711376 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:41.711465 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:41.711661 (+ 196us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 03:58:41.732183 (+ 20522us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43841/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 03:58:41.732373 (+ 190us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36250: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43841/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":45}
W20260926 03:58:41.732501 5688 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36250: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43841/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260926 03:58:41.732762 5713 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:58:41.709162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39683 (local address 127.0.0.1:36250)
0926 03:58:41.709364 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:41.709373 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:41.709449 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:41.709650 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:41.709651 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:41.709652 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0926 03:58:41.709718 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:41.709720 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:41.710423 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:41.710424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:41.711043 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:41.711045 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:41.711467 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:41.711476 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:41.711538 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:41.711544 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 03:58:41.732400 (+ 20856us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43841/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 03:58:41.732532 (+ 132us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:39683: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43841/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":115,"threads_started":1}
I20260926 03:58:41.733117 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 5617
2026-09-26T03:58:41Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (240 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260926 03:58:41.771302 392 mini_oidc.cc:152] Starting JWKS server
I20260926 03:58:41.772012 392 webserver.cc:460] Webserver started at https://127.0.0.1:34207/ using document root <none> and password file <none>
I20260926 03:58:41.772089 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 03:58:41.772256 392 webserver.cc:460] Webserver started at http://127.0.0.1:34651/ using document root <none> and password file <none>
2026-09-26T03:58:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:58:41Z Disabled control of system clock
I20260926 03:58:41.779632 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790395095474643-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790395095474643-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790395095474643-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41059
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41059
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34207/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 03:58:41.858273 5720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:58:41.858510 5720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:58:41.858543 5720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:58:41.859660 5720 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:58:41.859725 5720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:58:41.859748 5720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:58:41.859781 5720 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:58:41.859828 5720 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 03:58:41.859858 5720 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 03:58:41.859884 5720 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34207/jwks.json
I20260926 03:58:41.861284 5720 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790395095474643-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790395095474643-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41059
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34207/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790395095474643-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:58:41.861541 5720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:58:41.861814 5720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:58:41.866541 5726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:41.866525 5728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:41.866569 5720 server_base.cc:1034] running on GCE node
W20260926 03:58:41.866513 5725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:41.866978 5720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:58:41.867306 5720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:58:41.868500 5720 hybrid_clock.cc:648] HybridClock initialized: now 1790395121868481 us; error 31 us; skew 500 ppm
I20260926 03:58:41.869782 5720 webserver.cc:460] Webserver started at http://127.0.98.62:41329/ using document root <none> and password file <none>
I20260926 03:58:41.870013 5720 fs_manager.cc:359] Metadata directory not provided
I20260926 03:58:41.870082 5720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:58:41.870231 5720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:58:41.871268 5720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790395095474643-392-0/minicluster-data/master-0/data/instance:
uuid: "8fda6bd76c2b4287849b274541f01f58"
format_stamp: "Formatted at 2026-09-26 03:58:41 on dist-test-slave-nhpw"
server_key: "dad02197f7bf9a3dec2d7f50486fa0af"
server_key_iv: "3f3a0866dc09e04dd8b0097c44b17dec"
server_key_version: "encryptionkey@0"
I20260926 03:58:41.871627 5720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790395095474643-392-0/minicluster-data/master-0/wal/instance:
uuid: "8fda6bd76c2b4287849b274541f01f58"
format_stamp: "Formatted at 2026-09-26 03:58:41 on dist-test-slave-nhpw"
server_key: "dad02197f7bf9a3dec2d7f50486fa0af"
server_key_iv: "3f3a0866dc09e04dd8b0097c44b17dec"
server_key_version: "encryptionkey@0"
I20260926 03:58:41.872972 5720 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 03:58:41.873688 5734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:41.873852 5720 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 03:58:41.873948 5720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790395095474643-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790395095474643-392-0/minicluster-data/master-0/wal
uuid: "8fda6bd76c2b4287849b274541f01f58"
format_stamp: "Formatted at 2026-09-26 03:58:41 on dist-test-slave-nhpw"
server_key: "dad02197f7bf9a3dec2d7f50486fa0af"
server_key_iv: "3f3a0866dc09e04dd8b0097c44b17dec"
server_key_version: "encryptionkey@0"
I20260926 03:58:41.874035 5720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790395095474643-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790395095474643-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790395095474643-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:58:41.884775 5720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:58:41.885028 5720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:58:41.885183 5720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:58:41.888785 5720 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41059
I20260926 03:58:41.888846 5786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41059 every 8 connection(s)
I20260926 03:58:41.889160 5720 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
I20260926 03:58:41.889791 5787 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:58:41.892308 5787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fda6bd76c2b4287849b274541f01f58: Bootstrap starting.
I20260926 03:58:41.892930 5787 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8fda6bd76c2b4287849b274541f01f58: Neither blocks nor log segments found. Creating new log.
I20260926 03:58:41.893213 5787 log.cc:824] T 00000000000000000000000000000000 P 8fda6bd76c2b4287849b274541f01f58: Log is configured to *not* fsync() on all Append() calls
I20260926 03:58:41.893731 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 5720
I20260926 03:58:41.893841 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790395095474643-392-0/minicluster-data/master-0/wal/instance
I20260926 03:58:41.894001 5787 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8fda6bd76c2b4287849b274541f01f58: No bootstrap required, opened a new log
I20260926 03:58:41.894011 392 external_mini_cluster.cc:1428] Setting key f0fa0bbddd95b017c607557a62458a85
I20260926 03:58:41.896469 5787 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8fda6bd76c2b4287849b274541f01f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fda6bd76c2b4287849b274541f01f58" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41059 } }
I20260926 03:58:41.896685 5787 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8fda6bd76c2b4287849b274541f01f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:58:41.896736 5787 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8fda6bd76c2b4287849b274541f01f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8fda6bd76c2b4287849b274541f01f58, State: Initialized, Role: FOLLOWER
I20260926 03:58:41.896862 5787 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8fda6bd76c2b4287849b274541f01f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fda6bd76c2b4287849b274541f01f58" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41059 } }
I20260926 03:58:41.896935 5787 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8fda6bd76c2b4287849b274541f01f58 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:58:41.896965 5787 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8fda6bd76c2b4287849b274541f01f58 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:58:41.897007 5787 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8fda6bd76c2b4287849b274541f01f58 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:58:41.897655 5787 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8fda6bd76c2b4287849b274541f01f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fda6bd76c2b4287849b274541f01f58" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41059 } }
I20260926 03:58:41.897753 5787 leader_election.cc:302] T 00000000000000000000000000000000 P 8fda6bd76c2b4287849b274541f01f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8fda6bd76c2b4287849b274541f01f58; no voters:
I20260926 03:58:41.897925 5787 leader_election.cc:288] T 00000000000000000000000000000000 P 8fda6bd76c2b4287849b274541f01f58 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:58:41.897953 5792 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8fda6bd76c2b4287849b274541f01f58 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:58:41.898056 5792 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8fda6bd76c2b4287849b274541f01f58 [term 1 LEADER]: Becoming Leader. State: Replica: 8fda6bd76c2b4287849b274541f01f58, State: Running, Role: LEADER
I20260926 03:58:41.898178 5787 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8fda6bd76c2b4287849b274541f01f58 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:58:41.898195 5792 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8fda6bd76c2b4287849b274541f01f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fda6bd76c2b4287849b274541f01f58" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41059 } }
I20260926 03:58:41.898599 5794 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fda6bd76c2b4287849b274541f01f58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8fda6bd76c2b4287849b274541f01f58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fda6bd76c2b4287849b274541f01f58" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41059 } } }
I20260926 03:58:41.898702 5794 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fda6bd76c2b4287849b274541f01f58 [sys.catalog]: This master's current role is: LEADER
I20260926 03:58:41.898887 5794 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8fda6bd76c2b4287849b274541f01f58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8fda6bd76c2b4287849b274541f01f58. Latest consensus state: current_term: 1 leader_uuid: "8fda6bd76c2b4287849b274541f01f58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8fda6bd76c2b4287849b274541f01f58" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41059 } } }
I20260926 03:58:41.898979 5794 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8fda6bd76c2b4287849b274541f01f58 [sys.catalog]: This master's current role is: LEADER
W20260926 03:58:41.900035 5807 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8fda6bd76c2b4287849b274541f01f58: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260926 03:58:41.900055 5791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:41.894537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45638 (local address 127.0.98.62:41059)
0926 03:58:41.894732 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:41.894748 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:41.894772 (+ 24us) server_negotiation.cc:407] Connection header received
0926 03:58:41.895245 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:41.895251 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:41.895303 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:41.895410 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:41.895638 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:41.896492 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:41.897607 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:41.898233 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:41.898642 (+ 409us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:41.899081 (+ 439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:41.899085 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:41.899086 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:58:41.899306 (+ 220us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:41.899353 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:58:41.899367 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:41.899426 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:41.899771 (+ 345us) server_negotiation.cc:299] Negotiation successful
0926 03:58:41.899852 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":86,"threads_started":1}
W20260926 03:58:41.900173 5807 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 03:58:41.900245 5801 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:58:41.900380 5801 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:58:41.901078 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:58:41.901883 5801 catalog_manager.cc:1347] Generated new cluster ID: e31dc0dc692c4885bb201854958b6969
I20260926 03:58:41.901935 5801 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:58:41.917963 5801 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:58:41.918464 5801 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:58:41.931059 5801 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8fda6bd76c2b4287849b274541f01f58: Generated new TSK 0
I20260926 03:58:41.931258 5801 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:58:41.949548 5791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:41.945999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45654 (local address 127.0.98.62:41059)
0926 03:58:41.946054 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:41.946056 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:41.946152 (+ 96us) server_negotiation.cc:407] Connection header received
0926 03:58:41.946227 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:41.946227 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:41.946238 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:41.946295 (+ 57us) server_negotiation.cc:226] Negotiated authn=JWT
0926 03:58:41.946450 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:41.946908 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:41.948088 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:41.948266 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:41.948314 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:41.948505 (+ 191us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 03:58:41.949330 (+ 825us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34207/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 03:58:41.949474 (+ 144us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45654: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34207/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":29}
W20260926 03:58:41.949608 5791 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45654: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34207/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260926 03:58:41.950351 5816 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:58:41.945920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41059 (local address 127.0.0.1:45654)
0926 03:58:41.946077 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:41.946089 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:41.946159 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:41.946320 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:41.946321 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:41.946322 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0926 03:58:41.946384 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:41.946386 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:41.947030 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:41.947031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:41.948005 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:41.948007 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:41.948340 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:41.948346 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:41.948406 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:41.948411 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 03:58:41.949437 (+ 1026us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34207/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 03:58:41.950224 (+ 787us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:41059: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34207/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":101,"threads_started":1}
I20260926 03:58:41.950960 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 5720
E20260926 03:58:41.961735 5817 webserver.cc:533] Webserver: sslize failed:
E20260926 03:58:41.961794 5817 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-26T03:58:41Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (217 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-26T03:58:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:58:41Z Disabled control of system clock
I20260926 03:58:41.996100 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790395095474643-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790395095474643-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790395095474643-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45433
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:46561
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45433
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790395095474643-392-0/testchainca.cert with env {}
W20260926 03:58:42.074398 5821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:58:42.074595 5821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:58:42.074617 5821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:58:42.075691 5821 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:58:42.075724 5821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:58:42.075737 5821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:58:42.075749 5821 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:58:42.075762 5821 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 03:58:42.075773 5821 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 03:58:42.075784 5821 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260926 03:58:42.077215 5821 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46561
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790395095474643-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790395095474643-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45433
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790395095474643-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790395095474643-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:58:42.077392 5821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:58:42.077538 5821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:58:42.082283 5827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:42.082263 5826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:42.082435 5821 server_base.cc:1034] running on GCE node
W20260926 03:58:42.082432 5829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:42.082805 5821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:58:42.083076 5821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:58:42.084276 5821 hybrid_clock.cc:648] HybridClock initialized: now 1790395122084262 us; error 29 us; skew 500 ppm
I20260926 03:58:42.085604 5821 webserver.cc:460] Webserver started at http://127.0.98.62:44749/ using document root <none> and password file <none>
I20260926 03:58:42.085873 5821 fs_manager.cc:359] Metadata directory not provided
I20260926 03:58:42.085973 5821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:58:42.086136 5821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:58:42.087218 5821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790395095474643-392-0/minicluster-data/master-0/data/instance:
uuid: "6288386c274043469440e21a20f71e09"
format_stamp: "Formatted at 2026-09-26 03:58:42 on dist-test-slave-nhpw"
server_key: "11841c084db0d8b99212d5fdbed0bf39"
server_key_iv: "7e5e00e9647e4d09c564789ab968f85a"
server_key_version: "encryptionkey@0"
I20260926 03:58:42.087600 5821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790395095474643-392-0/minicluster-data/master-0/wal/instance:
uuid: "6288386c274043469440e21a20f71e09"
format_stamp: "Formatted at 2026-09-26 03:58:42 on dist-test-slave-nhpw"
server_key: "11841c084db0d8b99212d5fdbed0bf39"
server_key_iv: "7e5e00e9647e4d09c564789ab968f85a"
server_key_version: "encryptionkey@0"
I20260926 03:58:42.089046 5821 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.002s
I20260926 03:58:42.089874 5835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:42.090056 5821 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 03:58:42.090165 5821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790395095474643-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790395095474643-392-0/minicluster-data/master-0/wal
uuid: "6288386c274043469440e21a20f71e09"
format_stamp: "Formatted at 2026-09-26 03:58:42 on dist-test-slave-nhpw"
server_key: "11841c084db0d8b99212d5fdbed0bf39"
server_key_iv: "7e5e00e9647e4d09c564789ab968f85a"
server_key_version: "encryptionkey@0"
I20260926 03:58:42.090256 5821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790395095474643-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790395095474643-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790395095474643-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:58:42.099318 5821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:58:42.099669 5821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:58:42.099826 5821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:58:42.103993 5821 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45433
I20260926 03:58:42.104020 5887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45433 every 8 connection(s)
I20260926 03:58:42.104369 5821 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
I20260926 03:58:42.105275 5888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:58:42.107430 5888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6288386c274043469440e21a20f71e09: Bootstrap starting.
I20260926 03:58:42.107990 5888 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6288386c274043469440e21a20f71e09: Neither blocks nor log segments found. Creating new log.
I20260926 03:58:42.108271 5888 log.cc:824] T 00000000000000000000000000000000 P 6288386c274043469440e21a20f71e09: Log is configured to *not* fsync() on all Append() calls
I20260926 03:58:42.109043 5888 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6288386c274043469440e21a20f71e09: No bootstrap required, opened a new log
I20260926 03:58:42.109863 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 5821
I20260926 03:58:42.110008 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790395095474643-392-0/minicluster-data/master-0/wal/instance
I20260926 03:58:42.110201 392 external_mini_cluster.cc:1428] Setting key 3bae3622679af293b838ffd794fa9513
I20260926 03:58:42.110915 5888 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6288386c274043469440e21a20f71e09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6288386c274043469440e21a20f71e09" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45433 } }
I20260926 03:58:42.111068 5888 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6288386c274043469440e21a20f71e09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:58:42.111114 5888 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6288386c274043469440e21a20f71e09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6288386c274043469440e21a20f71e09, State: Initialized, Role: FOLLOWER
I20260926 03:58:42.111248 5888 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6288386c274043469440e21a20f71e09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6288386c274043469440e21a20f71e09" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45433 } }
I20260926 03:58:42.111343 5888 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6288386c274043469440e21a20f71e09 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:58:42.111382 5888 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6288386c274043469440e21a20f71e09 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:58:42.111428 5888 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6288386c274043469440e21a20f71e09 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:58:42.112160 5888 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6288386c274043469440e21a20f71e09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6288386c274043469440e21a20f71e09" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45433 } }
I20260926 03:58:42.112267 5888 leader_election.cc:302] T 00000000000000000000000000000000 P 6288386c274043469440e21a20f71e09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6288386c274043469440e21a20f71e09; no voters:
I20260926 03:58:42.112412 5888 leader_election.cc:288] T 00000000000000000000000000000000 P 6288386c274043469440e21a20f71e09 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:58:42.112490 5893 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6288386c274043469440e21a20f71e09 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:58:42.112624 5893 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6288386c274043469440e21a20f71e09 [term 1 LEADER]: Becoming Leader. State: Replica: 6288386c274043469440e21a20f71e09, State: Running, Role: LEADER
I20260926 03:58:42.112695 5888 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6288386c274043469440e21a20f71e09 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:58:42.112736 5893 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6288386c274043469440e21a20f71e09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6288386c274043469440e21a20f71e09" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45433 } }
I20260926 03:58:42.113057 5893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6288386c274043469440e21a20f71e09 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6288386c274043469440e21a20f71e09. Latest consensus state: current_term: 1 leader_uuid: "6288386c274043469440e21a20f71e09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6288386c274043469440e21a20f71e09" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45433 } } }
I20260926 03:58:42.113150 5893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6288386c274043469440e21a20f71e09 [sys.catalog]: This master's current role is: LEADER
I20260926 03:58:42.113070 5894 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6288386c274043469440e21a20f71e09 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6288386c274043469440e21a20f71e09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6288386c274043469440e21a20f71e09" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45433 } } }
I20260926 03:58:42.113271 5894 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6288386c274043469440e21a20f71e09 [sys.catalog]: This master's current role is: LEADER
I20260926 03:58:42.113344 5900 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:58:42.113840 5900 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:58:42.115381 5910 mvcc.cc:205] Tried to move back new op lower bound from 7333458420182446080 to 7333458420174131200. Current Snapshot: MvccSnapshot[applied={T|T < 7333458420182446080}]
I20260926 03:58:42.116485 5892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:42.110709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39876 (local address 127.0.98.62:45433)
0926 03:58:42.110938 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:42.110946 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:42.110969 (+ 23us) server_negotiation.cc:407] Connection header received
0926 03:58:42.111060 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:42.111065 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:42.111147 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:42.111296 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:42.111528 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:42.113324 (+ 1796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:42.114405 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:42.114956 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:42.115089 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:42.115816 (+ 727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:42.115820 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:42.115820 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:58:42.115989 (+ 169us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:42.116020 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:58:42.116030 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:42.116085 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:42.116183 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 03:58:42.116276 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":98,"threads_started":1}
I20260926 03:58:42.117014 5900 catalog_manager.cc:1347] Generated new cluster ID: 4706835a09094334a7f654bb2a81db82
I20260926 03:58:42.117089 5900 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:58:42.117447 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 03:58:42.129144 5900 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:58:42.129647 5900 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:58:42.138933 5900 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6288386c274043469440e21a20f71e09: Generated new TSK 0
I20260926 03:58:42.139124 5900 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:58:42.574908 5892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:42.149161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39884 (local address 127.0.98.62:45433)
0926 03:58:42.149206 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:42.149208 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:42.149273 (+ 65us) server_negotiation.cc:407] Connection header received
0926 03:58:42.149352 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:42.149353 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:42.149366 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:42.149435 (+ 69us) server_negotiation.cc:226] Negotiated authn=JWT
0926 03:58:42.149559 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:42.150345 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:42.151148 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:42.151454 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:42.151526 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:42.151685 (+ 159us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 03:58:42.574683 (+422998us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 03:58:42.574845 (+ 162us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39884: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":23}
W20260926 03:58:42.574975 5892 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39884: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260926 03:58:42.575139 5917 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:58:42.149049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45433 (local address 127.0.0.1:39884)
0926 03:58:42.149211 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:42.149221 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:42.149285 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:42.149441 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:42.149442 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:42.149443 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0926 03:58:42.149494 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:42.149496 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:42.150460 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:42.150461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:42.151063 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:42.151065 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:42.151539 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:42.151543 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:42.151586 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:42.151590 (+ 4us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 03:58:42.574828 (+423238us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 03:58:42.574962 (+ 134us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:45433: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":112,"threads_started":1}
I20260926 03:58:42.575656 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 5821
2026-09-26T03:58:42Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (604 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (89 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (85 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:58:42 dist-test-slave-nhpw krb5kdc[5929](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:58:42 dist-test-slave-nhpw krb5kdc[5929](info): set up 2 sockets
Sep 26 03:58:42 dist-test-slave-nhpw krb5kdc[5929](info): commencing operation
krb5kdc: starting...
W20260926 03:58:43.785003 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/krb5kdc/test-user.keytab.
Sep 26 03:58:43 dist-test-slave-nhpw krb5kdc[5929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:58:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:58:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 03:58:43.903961 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33075
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33075
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:58:43.983825 5942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:58:43.984020 5942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:58:43.984042 5942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:58:43.985131 5942 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:58:43.985163 5942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:58:43.985177 5942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:58:43.985190 5942 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:58:43.985203 5942 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:58:43.986629 5942 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33075
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5942
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:58:43.986802 5942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:58:43.986966 5942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 03:58:43.991824 5942 server_base.cc:1034] running on GCE node
W20260926 03:58:43.991776 5950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:43.991799 5948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:43.991789 5947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:43.992208 5942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:58:43.992548 5942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:58:43.993716 5942 hybrid_clock.cc:648] HybridClock initialized: now 1790395123993696 us; error 32 us; skew 500 ppm
Sep 26 03:58:43 dist-test-slave-nhpw krb5kdc[5929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:58:43.996421 5942 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 03:58:43.996923 5942 webserver.cc:460] Webserver started at http://127.0.98.62:43569/ using document root <none> and password file <none>
I20260926 03:58:43.997161 5942 fs_manager.cc:359] Metadata directory not provided
I20260926 03:58:43.997229 5942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:58:43.997362 5942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:58:43.998411 5942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/minicluster-data/master-0/data/instance:
uuid: "f14acadc01b44407a7e64139565bd0b2"
format_stamp: "Formatted at 2026-09-26 03:58:43 on dist-test-slave-nhpw"
server_key: "0888152f1180b2701beaa3bfa95552c3"
server_key_iv: "59cb0d33dca4d044d52ae141f6f55b72"
server_key_version: "encryptionkey@0"
I20260926 03:58:43.998782 5942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/minicluster-data/master-0/wal/instance:
uuid: "f14acadc01b44407a7e64139565bd0b2"
format_stamp: "Formatted at 2026-09-26 03:58:43 on dist-test-slave-nhpw"
server_key: "0888152f1180b2701beaa3bfa95552c3"
server_key_iv: "59cb0d33dca4d044d52ae141f6f55b72"
server_key_version: "encryptionkey@0"
I20260926 03:58:44.000095 5942 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 03:58:44.000883 5957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:44.001168 5942 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 03:58:44.001268 5942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/minicluster-data/master-0/wal
uuid: "f14acadc01b44407a7e64139565bd0b2"
format_stamp: "Formatted at 2026-09-26 03:58:43 on dist-test-slave-nhpw"
server_key: "0888152f1180b2701beaa3bfa95552c3"
server_key_iv: "59cb0d33dca4d044d52ae141f6f55b72"
server_key_version: "encryptionkey@0"
I20260926 03:58:44.001349 5942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:58:44.018679 5942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:58:44.020573 5942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:58:44.020761 5942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:58:44.024739 5942 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33075
I20260926 03:58:44.024775 6009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33075 every 8 connection(s)
I20260926 03:58:44.025276 5942 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
I20260926 03:58:44.025890 6010 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:58:44.028013 6010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f14acadc01b44407a7e64139565bd0b2: Bootstrap starting.
I20260926 03:58:44.028541 6010 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f14acadc01b44407a7e64139565bd0b2: Neither blocks nor log segments found. Creating new log.
I20260926 03:58:44.028553 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 5942
I20260926 03:58:44.028689 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/minicluster-data/master-0/wal/instance
I20260926 03:58:44.028879 6010 log.cc:824] T 00000000000000000000000000000000 P f14acadc01b44407a7e64139565bd0b2: Log is configured to *not* fsync() on all Append() calls
I20260926 03:58:44.028914 392 external_mini_cluster.cc:1428] Setting key 22a23f053baa985a31c08995837f78e9
I20260926 03:58:44.029631 6010 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f14acadc01b44407a7e64139565bd0b2: No bootstrap required, opened a new log
I20260926 03:58:44.031694 6010 raft_consensus.cc:348] T 00000000000000000000000000000000 P f14acadc01b44407a7e64139565bd0b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f14acadc01b44407a7e64139565bd0b2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33075 } }
I20260926 03:58:44.031847 6010 raft_consensus.cc:374] T 00000000000000000000000000000000 P f14acadc01b44407a7e64139565bd0b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:58:44.031895 6010 raft_consensus.cc:729] T 00000000000000000000000000000000 P f14acadc01b44407a7e64139565bd0b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f14acadc01b44407a7e64139565bd0b2, State: Initialized, Role: FOLLOWER
I20260926 03:58:44.032022 6010 consensus_queue.cc:260] T 00000000000000000000000000000000 P f14acadc01b44407a7e64139565bd0b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f14acadc01b44407a7e64139565bd0b2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33075 } }
I20260926 03:58:44.032090 6010 raft_consensus.cc:388] T 00000000000000000000000000000000 P f14acadc01b44407a7e64139565bd0b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:58:44.032112 6010 raft_consensus.cc:482] T 00000000000000000000000000000000 P f14acadc01b44407a7e64139565bd0b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:58:44.032177 6010 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f14acadc01b44407a7e64139565bd0b2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:58:44.032757 6010 raft_consensus.cc:504] T 00000000000000000000000000000000 P f14acadc01b44407a7e64139565bd0b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f14acadc01b44407a7e64139565bd0b2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33075 } }
I20260926 03:58:44.032868 6010 leader_election.cc:302] T 00000000000000000000000000000000 P f14acadc01b44407a7e64139565bd0b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f14acadc01b44407a7e64139565bd0b2; no voters:
I20260926 03:58:44.033018 6010 leader_election.cc:288] T 00000000000000000000000000000000 P f14acadc01b44407a7e64139565bd0b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:58:44.033064 6015 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f14acadc01b44407a7e64139565bd0b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:58:44.033222 6015 raft_consensus.cc:686] T 00000000000000000000000000000000 P f14acadc01b44407a7e64139565bd0b2 [term 1 LEADER]: Becoming Leader. State: Replica: f14acadc01b44407a7e64139565bd0b2, State: Running, Role: LEADER
I20260926 03:58:44.033288 6010 sys_catalog.cc:564] T 00000000000000000000000000000000 P f14acadc01b44407a7e64139565bd0b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:58:44.033353 6015 consensus_queue.cc:237] T 00000000000000000000000000000000 P f14acadc01b44407a7e64139565bd0b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f14acadc01b44407a7e64139565bd0b2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33075 } }
I20260926 03:58:44.033871 6016 sys_catalog.cc:455] T 00000000000000000000000000000000 P f14acadc01b44407a7e64139565bd0b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f14acadc01b44407a7e64139565bd0b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f14acadc01b44407a7e64139565bd0b2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33075 } } }
I20260926 03:58:44.033887 6017 sys_catalog.cc:455] T 00000000000000000000000000000000 P f14acadc01b44407a7e64139565bd0b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f14acadc01b44407a7e64139565bd0b2. Latest consensus state: current_term: 1 leader_uuid: "f14acadc01b44407a7e64139565bd0b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f14acadc01b44407a7e64139565bd0b2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33075 } } }
I20260926 03:58:44.033998 6016 sys_catalog.cc:458] T 00000000000000000000000000000000 P f14acadc01b44407a7e64139565bd0b2 [sys.catalog]: This master's current role is: LEADER
I20260926 03:58:44.034008 6017 sys_catalog.cc:458] T 00000000000000000000000000000000 P f14acadc01b44407a7e64139565bd0b2 [sys.catalog]: This master's current role is: LEADER
I20260926 03:58:44.034195 6024 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:58:44.034685 6024 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 03:58:44 dist-test-slave-nhpw krb5kdc[5929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395123, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 03:58:44.036415 6032 mvcc.cc:205] Tried to move back new op lower bound from 7333458428049551360 to 7333458428041261056. Current Snapshot: MvccSnapshot[applied={T|T < 7333458428049551360}]
I20260926 03:58:44.036896 6024 catalog_manager.cc:1347] Generated new cluster ID: 57b0409ede2f486db61b4e059de76d8f
I20260926 03:58:44.036952 6024 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:58:44.039669 6014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:44.029406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34704 (local address 127.0.98.62:33075)
0926 03:58:44.029592 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:44.029602 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:44.029633 (+ 31us) server_negotiation.cc:407] Connection header received
0926 03:58:44.030091 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:44.030098 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:44.030142 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:44.030288 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:44.030904 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:44.031752 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:44.032665 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:44.033212 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:44.033587 (+ 375us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:44.036291 (+ 2704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:44.036296 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:44.036296 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:44.036309 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:44.038145 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:44.038540 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:44.038543 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:44.038543 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:44.038640 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:44.038858 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:44.038859 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:44.038859 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:44.039109 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:44.039294 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:44.039361 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 03:58:44.039497 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":87,"threads_started":1}
I20260926 03:58:44.060667 6024 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:58:44.061379 6024 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:58:44.068006 6024 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f14acadc01b44407a7e64139565bd0b2: Generated new TSK 0
I20260926 03:58:44.068179 6024 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 03:58:44.096187 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33075
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:58:44.177582 6036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:58:44.177779 6036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:58:44.177801 6036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:58:44.178953 6036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:58:44.178993 6036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:58:44.179019 6036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 03:58:44.180527 6036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33075
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6036
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:58:44.180779 6036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:58:44.181018 6036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:58:44.186007 6041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:44.185992 6042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:44.186111 6036 server_base.cc:1034] running on GCE node
W20260926 03:58:44.186053 6044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:44.186436 6036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:58:44.186673 6036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:58:44.187839 6036 hybrid_clock.cc:648] HybridClock initialized: now 1790395124187836 us; error 37 us; skew 500 ppm
Sep 26 03:58:44 dist-test-slave-nhpw krb5kdc[5929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395124, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:58:44.190407 6036 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 03:58:44.190867 6036 webserver.cc:460] Webserver started at http://127.0.98.1:43175/ using document root <none> and password file <none>
I20260926 03:58:44.191102 6036 fs_manager.cc:359] Metadata directory not provided
I20260926 03:58:44.191166 6036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:58:44.191296 6036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:58:44.192296 6036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/minicluster-data/ts-0/data/instance:
uuid: "79911a1b3ed447d5b7957feb0b2b6d04"
format_stamp: "Formatted at 2026-09-26 03:58:44 on dist-test-slave-nhpw"
server_key: "439a70855eb219e0ba5f35cc713f743e"
server_key_iv: "19888d309f39fd178dd470e1b5746046"
server_key_version: "encryptionkey@0"
I20260926 03:58:44.192711 6036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/minicluster-data/ts-0/wal/instance:
uuid: "79911a1b3ed447d5b7957feb0b2b6d04"
format_stamp: "Formatted at 2026-09-26 03:58:44 on dist-test-slave-nhpw"
server_key: "439a70855eb219e0ba5f35cc713f743e"
server_key_iv: "19888d309f39fd178dd470e1b5746046"
server_key_version: "encryptionkey@0"
I20260926 03:58:44.193994 6036 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 03:58:44.194712 6051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:44.194870 6036 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 03:58:44.194965 6036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/minicluster-data/ts-0/wal
uuid: "79911a1b3ed447d5b7957feb0b2b6d04"
format_stamp: "Formatted at 2026-09-26 03:58:44 on dist-test-slave-nhpw"
server_key: "439a70855eb219e0ba5f35cc713f743e"
server_key_iv: "19888d309f39fd178dd470e1b5746046"
server_key_version: "encryptionkey@0"
I20260926 03:58:44.195051 6036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:58:44.224774 6036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:58:44.227190 6036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:58:44.227357 6036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:58:44.227607 6036 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:58:44.227933 6036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:58:44.227986 6036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:44.228027 6036 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:58:44.228055 6036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:44.233884 6036 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37107
I20260926 03:58:44.233991 6164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37107 every 8 connection(s)
I20260926 03:58:44.234309 6036 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
Sep 26 03:58:44 dist-test-slave-nhpw krb5kdc[5929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395124, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 03:58:44.241048 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 6036
I20260926 03:58:44.241166 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/minicluster-data/ts-0/wal/instance
I20260926 03:58:44.241405 392 external_mini_cluster.cc:1428] Setting key 69b05aaf749833ca90751fe65b155e14
I20260926 03:58:44.244052 6014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:44.235128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45809 (local address 127.0.98.62:33075)
0926 03:58:44.235195 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:44.235197 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:44.235391 (+ 194us) server_negotiation.cc:407] Connection header received
0926 03:58:44.236330 (+ 939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:44.236331 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:44.236340 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:44.236394 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:44.236988 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:44.237482 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:44.238437 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:44.238623 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:44.238674 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:44.240903 (+ 2229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:44.240905 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:44.240906 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:44.240913 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:44.242673 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:44.243076 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:44.243078 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:44.243078 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:44.243140 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:44.243410 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:44.243411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:44.243411 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:44.243538 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:44.243618 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:44.243870 (+ 252us) server_negotiation.cc:299] Negotiation successful
0926 03:58:44.243956 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 03:58:44.244269 6167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:44.235106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33075 (local address 127.0.98.1:45809)
0926 03:58:44.235264 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:44.235306 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:44.236243 (+ 937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:44.236466 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:44.236469 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:44.236732 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:44.236915 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:44.236920 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:44.237565 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:44.237566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:44.238360 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:44.238362 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:44.238711 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:44.238712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:44.238818 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:44.239357 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:44.239365 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:44.240802 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:58:44.242754 (+ 1952us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:44.242759 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:44.242761 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:44.242966 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:44.243207 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:44.243208 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:44.243208 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:44.243300 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:44.243608 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:58:44.243614 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:58:44.243743 (+ 129us) client_negotiation.cc:769] Sending connection context
0926 03:58:44.243851 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 03:58:44.244049 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":84,"threads_started":1}
I20260926 03:58:44.244733 6165 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33075
I20260926 03:58:44.244853 6165 heartbeater.cc:464] Registering TS with master...
I20260926 03:58:44.245023 6165 heartbeater.cc:511] Master 127.0.98.62:33075 requested a full tablet report, sending...
I20260926 03:58:44.245577 5974 ts_manager.cc:194] Registered new tserver with Master: 79911a1b3ed447d5b7957feb0b2b6d04 (127.0.98.1:37107)
I20260926 03:58:44.246565 5974 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45809
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 03:58:44.267271 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33075
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:58:44.347379 6170 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:58:44.347622 6170 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:58:44.347659 6170 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:58:44.348821 6170 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:58:44.348892 6170 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:58:44.348949 6170 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 03:58:44.350410 6170 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33075
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6170
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:58:44.350675 6170 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:58:44.350901 6170 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:58:44.355880 6178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:44.355875 6176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:44.355875 6175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:44.356122 6170 server_base.cc:1034] running on GCE node
I20260926 03:58:44.356316 6170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:58:44.356563 6170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:58:44.357733 6170 hybrid_clock.cc:648] HybridClock initialized: now 1790395124357711 us; error 35 us; skew 500 ppm
Sep 26 03:58:44 dist-test-slave-nhpw krb5kdc[5929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395124, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:58:44.360097 6170 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 03:58:44.360503 6170 webserver.cc:460] Webserver started at http://127.0.98.2:34073/ using document root <none> and password file <none>
I20260926 03:58:44.360785 6170 fs_manager.cc:359] Metadata directory not provided
I20260926 03:58:44.360853 6170 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:58:44.361007 6170 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:58:44.361987 6170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/minicluster-data/ts-1/data/instance:
uuid: "f29c0fb919f54e3b9e9949340ed25b11"
format_stamp: "Formatted at 2026-09-26 03:58:44 on dist-test-slave-nhpw"
server_key: "c61dca22fe37042d0a22511c5a879837"
server_key_iv: "895acd5f6bd7af5af8f33acbd7828321"
server_key_version: "encryptionkey@0"
I20260926 03:58:44.362342 6170 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f29c0fb919f54e3b9e9949340ed25b11"
format_stamp: "Formatted at 2026-09-26 03:58:44 on dist-test-slave-nhpw"
server_key: "c61dca22fe37042d0a22511c5a879837"
server_key_iv: "895acd5f6bd7af5af8f33acbd7828321"
server_key_version: "encryptionkey@0"
I20260926 03:58:44.363596 6170 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 03:58:44.364302 6185 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:44.364468 6170 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 03:58:44.364567 6170 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/minicluster-data/ts-1/wal
uuid: "f29c0fb919f54e3b9e9949340ed25b11"
format_stamp: "Formatted at 2026-09-26 03:58:44 on dist-test-slave-nhpw"
server_key: "c61dca22fe37042d0a22511c5a879837"
server_key_iv: "895acd5f6bd7af5af8f33acbd7828321"
server_key_version: "encryptionkey@0"
I20260926 03:58:44.364672 6170 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:58:44.384898 6170 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:58:44.387202 6170 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:58:44.387357 6170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:58:44.387593 6170 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:58:44.387894 6170 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:58:44.387948 6170 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:44.387988 6170 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:58:44.388017 6170 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:44.393879 6170 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39507
I20260926 03:58:44.393927 6298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39507 every 8 connection(s)
I20260926 03:58:44.394232 6170 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
Sep 26 03:58:44 dist-test-slave-nhpw krb5kdc[5929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395124, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 03:58:44.401853 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 6170
I20260926 03:58:44.401964 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/minicluster-data/ts-1/wal/instance
I20260926 03:58:44.402169 392 external_mini_cluster.cc:1428] Setting key ec37e008d41d2e0720087b3670adb21d
I20260926 03:58:44.403564 6014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:44.394962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58413 (local address 127.0.98.62:33075)
0926 03:58:44.395014 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:44.395016 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:44.395146 (+ 130us) server_negotiation.cc:407] Connection header received
0926 03:58:44.396079 (+ 933us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:44.396080 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:44.396090 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:44.396157 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:44.396725 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:44.397200 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:44.398299 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:44.398483 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:44.398546 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:44.400682 (+ 2136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:44.400684 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:44.400685 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:44.400691 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:44.402047 (+ 1356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:44.402510 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:44.402512 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:44.402512 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:44.402577 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:44.402864 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:44.402865 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:44.402865 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:44.403011 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:44.403084 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:44.403333 (+ 249us) server_negotiation.cc:299] Negotiation successful
0926 03:58:44.403421 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 03:58:44.403806 6301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:44.394884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33075 (local address 127.0.98.2:58413)
0926 03:58:44.395033 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:44.395075 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:44.395989 (+ 914us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:44.396212 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:44.396215 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:44.396449 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:44.396660 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:44.396665 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:44.397304 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:44.397305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:44.398179 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:44.398181 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:44.398560 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:44.398560 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:44.398657 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:44.399225 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:44.399233 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:44.400551 (+ 1318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:58:44.402123 (+ 1572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:44.402128 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:44.402130 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:44.402440 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:44.402644 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:44.402645 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:44.402645 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:44.402775 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:44.403092 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:58:44.403096 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:58:44.403208 (+ 112us) client_negotiation.cc:769] Sending connection context
0926 03:58:44.403293 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 03:58:44.403555 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":92,"threads_started":1}
I20260926 03:58:44.404397 6299 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33075
I20260926 03:58:44.404496 6299 heartbeater.cc:464] Registering TS with master...
I20260926 03:58:44.404722 6299 heartbeater.cc:511] Master 127.0.98.62:33075 requested a full tablet report, sending...
I20260926 03:58:44.405133 5974 ts_manager.cc:194] Registered new tserver with Master: f29c0fb919f54e3b9e9949340ed25b11 (127.0.98.2:39507)
I20260926 03:58:44.405592 5974 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58413
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 03:58:44.427292 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33075
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46211
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:58:44.515530 6304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:58:44.515756 6304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:58:44.515781 6304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:58:44.517273 6304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:58:44.517320 6304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:58:44.517352 6304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 03:58:44.519058 6304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46211
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33075
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:58:44.519316 6304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:58:44.519569 6304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:58:44.524919 6310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:44.524919 6309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:44.524940 6312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:44.525008 6304 server_base.cc:1034] running on GCE node
I20260926 03:58:44.525384 6304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:58:44.525646 6304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:58:44.526844 6304 hybrid_clock.cc:648] HybridClock initialized: now 1790395124526831 us; error 33 us; skew 500 ppm
Sep 26 03:58:44 dist-test-slave-nhpw krb5kdc[5929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395124, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:58:44.529502 6304 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 03:58:44.529970 6304 webserver.cc:460] Webserver started at http://127.0.98.3:45403/ using document root <none> and password file <none>
I20260926 03:58:44.530189 6304 fs_manager.cc:359] Metadata directory not provided
I20260926 03:58:44.530257 6304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:58:44.530408 6304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:58:44.531488 6304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/minicluster-data/ts-2/data/instance:
uuid: "fbbbba857e4b484a9a89dd41c3e39e06"
format_stamp: "Formatted at 2026-09-26 03:58:44 on dist-test-slave-nhpw"
server_key: "d14e0296a4928089d0656080b756ace4"
server_key_iv: "4c1c925e38619ebbd0a23ab9a23a4a30"
server_key_version: "encryptionkey@0"
I20260926 03:58:44.531853 6304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/minicluster-data/ts-2/wal/instance:
uuid: "fbbbba857e4b484a9a89dd41c3e39e06"
format_stamp: "Formatted at 2026-09-26 03:58:44 on dist-test-slave-nhpw"
server_key: "d14e0296a4928089d0656080b756ace4"
server_key_iv: "4c1c925e38619ebbd0a23ab9a23a4a30"
server_key_version: "encryptionkey@0"
I20260926 03:58:44.533253 6304 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 03:58:44.534107 6319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:44.534269 6304 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 03:58:44.534373 6304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/minicluster-data/ts-2/wal
uuid: "fbbbba857e4b484a9a89dd41c3e39e06"
format_stamp: "Formatted at 2026-09-26 03:58:44 on dist-test-slave-nhpw"
server_key: "d14e0296a4928089d0656080b756ace4"
server_key_iv: "4c1c925e38619ebbd0a23ab9a23a4a30"
server_key_version: "encryptionkey@0"
I20260926 03:58:44.534447 6304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:58:44.552791 6304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:58:44.555154 6304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:58:44.555318 6304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:58:44.555558 6304 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:58:44.555882 6304 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:58:44.555945 6304 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:44.555986 6304 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:58:44.556015 6304 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:44.562072 6304 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37497
I20260926 03:58:44.562116 6432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37497 every 8 connection(s)
I20260926 03:58:44.562430 6304 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
Sep 26 03:58:44 dist-test-slave-nhpw krb5kdc[5929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395124, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 03:58:44.571626 6014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:44.563231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59307 (local address 127.0.98.62:33075)
0926 03:58:44.563279 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:44.563281 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:44.563544 (+ 263us) server_negotiation.cc:407] Connection header received
0926 03:58:44.564437 (+ 893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:44.564439 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:44.564449 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:44.564504 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:44.565165 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:44.565649 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:44.566641 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:44.566827 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:44.566883 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:44.569222 (+ 2339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:44.569224 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:44.569224 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:44.569230 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:44.570378 (+ 1148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:44.570757 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:44.570759 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:44.570759 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:44.570837 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:44.571106 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:44.571107 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:44.571107 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:44.571209 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:44.571260 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:44.571495 (+ 235us) server_negotiation.cc:299] Negotiation successful
0926 03:58:44.571560 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 03:58:44.571849 6435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:44.563266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33075 (local address 127.0.98.3:59307)
0926 03:58:44.563418 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:44.563456 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:44.564347 (+ 891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:44.564556 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:44.564559 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:44.564883 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:44.565082 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:44.565087 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:44.565732 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:44.565733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:44.566564 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:44.566567 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:44.566926 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:44.566927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:44.567058 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:44.567670 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:44.567678 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:44.569138 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:58:44.570463 (+ 1325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:44.570468 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:44.570470 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:44.570672 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:44.570925 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:44.570926 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:44.570926 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:44.571039 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:44.571266 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:58:44.571271 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:58:44.571381 (+ 110us) client_negotiation.cc:769] Sending connection context
0926 03:58:44.571475 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 03:58:44.571595 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":90,"threads_started":1}
I20260926 03:58:44.572270 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 6304
I20260926 03:58:44.572357 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/minicluster-data/ts-2/wal/instance
I20260926 03:58:44.572465 6433 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33075
I20260926 03:58:44.572561 6433 heartbeater.cc:464] Registering TS with master...
I20260926 03:58:44.572561 392 external_mini_cluster.cc:1428] Setting key fb6428bc8eb8aaa3fa4f4aaa9d7c86ce
I20260926 03:58:44.572773 6433 heartbeater.cc:511] Master 127.0.98.62:33075 requested a full tablet report, sending...
I20260926 03:58:44.573166 5974 ts_manager.cc:194] Registered new tserver with Master: fbbbba857e4b484a9a89dd41c3e39e06 (127.0.98.3:37497)
I20260926 03:58:44.573458 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:58:44.573587 5974 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59307
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/krb5kdc/test-admin.keytab.
Sep 26 03:58:44 dist-test-slave-nhpw krb5kdc[5929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:58:44.589699 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260926 03:58:44.591200 6440 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:58:44.590474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39532 (local address 127.0.98.1:37107)
0926 03:58:44.590642 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:44.590646 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:44.590656 (+ 10us) server_negotiation.cc:407] Connection header received
0926 03:58:44.590680 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:44.590682 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:44.590695 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:44.590766 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:44.591097 (+ 331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39532: BlockingRecv error: recv got EOF from 127.0.0.1:39532 (error 108)
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":87,"threads_started":1}
W20260926 03:58:44.591317 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/krb5kdc/krb5cc' of type FILE
Sep 26 03:58:44 dist-test-slave-nhpw krb5kdc[5929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:58:44.592339 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 03:58:44 dist-test-slave-nhpw krb5kdc[5929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:58:44.593281 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 03:58:44 dist-test-slave-nhpw krb5kdc[5929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 03:58:44.600567 6440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:44.593814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39544 (local address 127.0.98.1:37107)
0926 03:58:44.593869 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:44.593870 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:44.593879 (+ 9us) server_negotiation.cc:407] Connection header received
0926 03:58:44.593949 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:44.593950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:44.593959 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:44.594005 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:44.594353 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:44.595057 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:44.595745 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:44.596049 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:44.596100 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:44.598211 (+ 2111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:44.598217 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:44.598217 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:44.598223 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:44.599554 (+ 1331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:44.599887 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:44.599890 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:44.599891 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:44.599948 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:44.600126 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:44.600127 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:44.600127 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:44.600257 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:44.600377 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:44.600430 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 03:58:44.600495 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
W20260926 03:58:44.602279 6440 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:58:44.601788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39554 (local address 127.0.98.1:37107)
0926 03:58:44.601841 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:44.601843 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:44.601927 (+ 84us) server_negotiation.cc:407] Connection header received
0926 03:58:44.601957 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:44.601958 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:44.601966 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:44.602001 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:44.602225 (+ 224us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39554: BlockingRecv error: recv got EOF from 127.0.0.1:39554 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260926 03:58:44.602425 392 init.cc:244] Error while opening credential cache '/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/krb5kdc/krb5cc)
Sep 26 03:58:44 dist-test-slave-nhpw krb5kdc[5929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:58:44.603358 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 03:58:44 dist-test-slave-nhpw krb5kdc[5929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:58:44.604218 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 03:58:44 dist-test-slave-nhpw krb5kdc[5929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 03:58:44.611111 6440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:44.604763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39564 (local address 127.0.98.1:37107)
0926 03:58:44.604804 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:44.604806 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:44.604863 (+ 57us) server_negotiation.cc:407] Connection header received
0926 03:58:44.604959 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:44.604960 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:44.604972 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:44.605070 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:44.605436 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:44.605901 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:44.606616 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:44.606916 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:44.606972 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:44.608906 (+ 1934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:44.608907 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:44.608908 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:44.608915 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:44.610006 (+ 1091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:44.610363 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:44.610364 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:44.610364 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:44.610419 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:44.610631 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:44.610632 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:44.610633 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:44.610725 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:44.610822 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:44.610942 (+ 120us) server_negotiation.cc:299] Negotiation successful
0926 03:58:44.611016 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260926 03:58:44.612563 6440 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:58:44.612121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39578 (local address 127.0.98.1:37107)
0926 03:58:44.612158 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:44.612160 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:44.612169 (+ 9us) server_negotiation.cc:407] Connection header received
0926 03:58:44.612224 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:44.612224 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:44.612231 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:44.612269 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:44.612512 (+ 243us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39578: BlockingRecv error: recv got EOF from 127.0.0.1:39578 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260926 03:58:44.612658 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790395095474643-392-0/krb5kdc/krb5cc' of type FILE
Sep 26 03:58:44 dist-test-slave-nhpw krb5kdc[5929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:58:44.613550 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 03:58:44 dist-test-slave-nhpw krb5kdc[5929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:58:44.614452 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 03:58:44 dist-test-slave-nhpw krb5kdc[5929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 03:58:44.621320 6440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:44.614926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39592 (local address 127.0.98.1:37107)
0926 03:58:44.614980 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:44.615011 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:44.615045 (+ 34us) server_negotiation.cc:407] Connection header received
0926 03:58:44.615060 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:44.615060 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:44.615067 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:44.615108 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:44.615487 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:44.615942 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:44.616827 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:44.617177 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:44.617266 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:44.619125 (+ 1859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:44.619126 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:44.619126 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:44.619133 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:44.620245 (+ 1112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:44.620604 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:44.620606 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:44.620606 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:44.620674 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:44.620862 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:44.620863 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:44.620863 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:44.620961 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:44.621057 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:44.621165 (+ 108us) server_negotiation.cc:299] Negotiation successful
0926 03:58:44.621234 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 03:58:44.621860 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 6036
I20260926 03:58:44.631034 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 6170
I20260926 03:58:44.636579 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 6304
I20260926 03:58:44.642223 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 5942
2026-09-26T03:58:44Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1892 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:58:44 dist-test-slave-nhpw krb5kdc[6453](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:58:44 dist-test-slave-nhpw krb5kdc[6453](info): set up 2 sockets
Sep 26 03:58:44 dist-test-slave-nhpw krb5kdc[6453](info): commencing operation
krb5kdc: starting...
W20260926 03:58:45.696946 392 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/krb5kdc/test-user.keytab.
Sep 26 03:58:45 dist-test-slave-nhpw krb5kdc[6453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:58:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:58:45Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260926 03:58:45.811082 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41421
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41421
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:58:45.889678 6466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:58:45.889868 6466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:58:45.889889 6466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:58:45.890925 6466 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:58:45.890959 6466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:58:45.890973 6466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:58:45.890985 6466 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:58:45.890997 6466 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:58:45.892377 6466 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41421
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:58:45.892539 6466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:58:45.892735 6466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:58:45.897498 6474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:45.897502 6471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:45.897680 6466 server_base.cc:1034] running on GCE node
W20260926 03:58:45.897519 6472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:45.897970 6466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:58:45.898242 6466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:58:45.899426 6466 hybrid_clock.cc:648] HybridClock initialized: now 1790395125899407 us; error 30 us; skew 500 ppm
Sep 26 03:58:45 dist-test-slave-nhpw krb5kdc[6453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395125, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:58:45.901940 6466 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260926 03:58:45.902395 6466 webserver.cc:460] Webserver started at http://127.0.98.62:41773/ using document root <none> and password file <none>
I20260926 03:58:45.902627 6466 fs_manager.cc:359] Metadata directory not provided
I20260926 03:58:45.902691 6466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:58:45.902823 6466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:58:45.903879 6466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/master-0/data/instance:
uuid: "0aeea5ec3a114f8c94fd2d2012fe312b"
format_stamp: "Formatted at 2026-09-26 03:58:45 on dist-test-slave-nhpw"
server_key: "cf163f9cb242ab756459c313eb918af1"
server_key_iv: "2b12d34caefe443421bdc57feb1ae74e"
server_key_version: "encryptionkey@0"
I20260926 03:58:45.904242 6466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/master-0/wal/instance:
uuid: "0aeea5ec3a114f8c94fd2d2012fe312b"
format_stamp: "Formatted at 2026-09-26 03:58:45 on dist-test-slave-nhpw"
server_key: "cf163f9cb242ab756459c313eb918af1"
server_key_iv: "2b12d34caefe443421bdc57feb1ae74e"
server_key_version: "encryptionkey@0"
I20260926 03:58:45.905675 6466 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 03:58:45.906435 6481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:45.906657 6466 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 03:58:45.906744 6466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/master-0/wal
uuid: "0aeea5ec3a114f8c94fd2d2012fe312b"
format_stamp: "Formatted at 2026-09-26 03:58:45 on dist-test-slave-nhpw"
server_key: "cf163f9cb242ab756459c313eb918af1"
server_key_iv: "2b12d34caefe443421bdc57feb1ae74e"
server_key_version: "encryptionkey@0"
I20260926 03:58:45.906796 6466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:58:45.926011 6466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:58:45.927834 6466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:58:45.927943 6466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:58:45.931802 6466 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41421
I20260926 03:58:45.932045 6533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41421 every 8 connection(s)
I20260926 03:58:45.932288 6466 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
I20260926 03:58:45.933290 6534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:58:45.935339 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 6466
I20260926 03:58:45.935449 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/master-0/wal/instance
I20260926 03:58:45.935616 6534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0aeea5ec3a114f8c94fd2d2012fe312b: Bootstrap starting.
I20260926 03:58:45.935655 392 external_mini_cluster.cc:1428] Setting key e53c15b69868815f4e73e939c1bba0db
I20260926 03:58:45.936334 6534 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0aeea5ec3a114f8c94fd2d2012fe312b: Neither blocks nor log segments found. Creating new log.
I20260926 03:58:45.936622 6534 log.cc:824] T 00000000000000000000000000000000 P 0aeea5ec3a114f8c94fd2d2012fe312b: Log is configured to *not* fsync() on all Append() calls
I20260926 03:58:45.937419 6534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0aeea5ec3a114f8c94fd2d2012fe312b: No bootstrap required, opened a new log
I20260926 03:58:45.939698 6534 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0aeea5ec3a114f8c94fd2d2012fe312b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aeea5ec3a114f8c94fd2d2012fe312b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41421 } }
I20260926 03:58:45.939862 6534 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0aeea5ec3a114f8c94fd2d2012fe312b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:58:45.939898 6534 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0aeea5ec3a114f8c94fd2d2012fe312b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0aeea5ec3a114f8c94fd2d2012fe312b, State: Initialized, Role: FOLLOWER
I20260926 03:58:45.940006 6534 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0aeea5ec3a114f8c94fd2d2012fe312b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aeea5ec3a114f8c94fd2d2012fe312b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41421 } }
I20260926 03:58:45.940070 6534 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0aeea5ec3a114f8c94fd2d2012fe312b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:58:45.940099 6534 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0aeea5ec3a114f8c94fd2d2012fe312b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:58:45.940151 6534 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0aeea5ec3a114f8c94fd2d2012fe312b [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:58:45.940830 6534 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0aeea5ec3a114f8c94fd2d2012fe312b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aeea5ec3a114f8c94fd2d2012fe312b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41421 } }
I20260926 03:58:45.940950 6534 leader_election.cc:302] T 00000000000000000000000000000000 P 0aeea5ec3a114f8c94fd2d2012fe312b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0aeea5ec3a114f8c94fd2d2012fe312b; no voters:
I20260926 03:58:45.941099 6534 leader_election.cc:288] T 00000000000000000000000000000000 P 0aeea5ec3a114f8c94fd2d2012fe312b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:58:45.941136 6539 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0aeea5ec3a114f8c94fd2d2012fe312b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:58:45.941285 6539 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0aeea5ec3a114f8c94fd2d2012fe312b [term 1 LEADER]: Becoming Leader. State: Replica: 0aeea5ec3a114f8c94fd2d2012fe312b, State: Running, Role: LEADER
I20260926 03:58:45.941359 6534 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0aeea5ec3a114f8c94fd2d2012fe312b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:58:45.941375 6539 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0aeea5ec3a114f8c94fd2d2012fe312b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aeea5ec3a114f8c94fd2d2012fe312b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41421 } }
I20260926 03:58:45.941747 6539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0aeea5ec3a114f8c94fd2d2012fe312b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0aeea5ec3a114f8c94fd2d2012fe312b. Latest consensus state: current_term: 1 leader_uuid: "0aeea5ec3a114f8c94fd2d2012fe312b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aeea5ec3a114f8c94fd2d2012fe312b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41421 } } }
I20260926 03:58:45.941838 6539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0aeea5ec3a114f8c94fd2d2012fe312b [sys.catalog]: This master's current role is: LEADER
I20260926 03:58:45.941758 6541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0aeea5ec3a114f8c94fd2d2012fe312b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0aeea5ec3a114f8c94fd2d2012fe312b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0aeea5ec3a114f8c94fd2d2012fe312b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41421 } } }
I20260926 03:58:45.942027 6546 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:58:45.942126 6541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0aeea5ec3a114f8c94fd2d2012fe312b [sys.catalog]: This master's current role is: LEADER
I20260926 03:58:45.942559 6546 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 03:58:45 dist-test-slave-nhpw krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 03:58:45.944221 6546 catalog_manager.cc:1347] Generated new cluster ID: a989010329ac4d63b6f8b327d43ccef5
I20260926 03:58:45.944271 6546 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:58:45.947381 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:45.936170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47536 (local address 127.0.98.62:41421)
0926 03:58:45.936322 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:45.936329 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:45.936354 (+ 25us) server_negotiation.cc:407] Connection header received
0926 03:58:45.936915 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:45.936922 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:45.936969 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:45.937088 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:45.937752 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:45.939123 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:45.939839 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:45.940442 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:45.940555 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:45.943392 (+ 2837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:45.943396 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:45.943396 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:45.943407 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:45.945378 (+ 1971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:45.945793 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:45.945796 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:45.945797 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:45.945895 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:45.946098 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:45.946099 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:45.946099 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:45.946377 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:45.946518 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:45.947022 (+ 504us) server_negotiation.cc:299] Negotiation successful
0926 03:58:45.947167 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":73,"threads_started":1}
I20260926 03:58:45.982986 6546 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:58:45.983544 6546 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:58:45.990715 6546 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0aeea5ec3a114f8c94fd2d2012fe312b: Generated new TSK 0
I20260926 03:58:45.990903 6546 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260926 03:58:46.023161 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41421
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:58:46.105093 6560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:58:46.105301 6560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:58:46.105326 6560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:58:46.106510 6560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:58:46.106549 6560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:58:46.106575 6560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 03:58:46.108161 6560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:58:46.108364 6560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:58:46.108564 6560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:58:46.114289 6566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:46.114315 6565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:46.114306 6568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:46.114565 6560 server_base.cc:1034] running on GCE node
I20260926 03:58:46.114742 6560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:58:46.114950 6560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:58:46.116130 6560 hybrid_clock.cc:648] HybridClock initialized: now 1790395126116122 us; error 43 us; skew 500 ppm
Sep 26 03:58:46 dist-test-slave-nhpw krb5kdc[6453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395126, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:58:46.118610 6560 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260926 03:58:46.119050 6560 webserver.cc:460] Webserver started at http://127.0.98.1:43513/ using document root <none> and password file <none>
I20260926 03:58:46.119292 6560 fs_manager.cc:359] Metadata directory not provided
I20260926 03:58:46.119369 6560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:58:46.119518 6560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:58:46.120537 6560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-0/data/instance:
uuid: "226f08ac9eb34274ae439b88cea9050c"
format_stamp: "Formatted at 2026-09-26 03:58:46 on dist-test-slave-nhpw"
server_key: "78f7b1ebb2c68787fbbbf09bbf0feeb9"
server_key_iv: "67d2f7c752cb906ace8a4ac5738b2a82"
server_key_version: "encryptionkey@0"
I20260926 03:58:46.120951 6560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-0/wal/instance:
uuid: "226f08ac9eb34274ae439b88cea9050c"
format_stamp: "Formatted at 2026-09-26 03:58:46 on dist-test-slave-nhpw"
server_key: "78f7b1ebb2c68787fbbbf09bbf0feeb9"
server_key_iv: "67d2f7c752cb906ace8a4ac5738b2a82"
server_key_version: "encryptionkey@0"
I20260926 03:58:46.122284 6560 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 03:58:46.123016 6575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:46.123175 6560 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 03:58:46.123279 6560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-0/wal
uuid: "226f08ac9eb34274ae439b88cea9050c"
format_stamp: "Formatted at 2026-09-26 03:58:46 on dist-test-slave-nhpw"
server_key: "78f7b1ebb2c68787fbbbf09bbf0feeb9"
server_key_iv: "67d2f7c752cb906ace8a4ac5738b2a82"
server_key_version: "encryptionkey@0"
I20260926 03:58:46.123364 6560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:58:46.143520 6560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:58:46.145373 6560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:58:46.145532 6560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:58:46.145751 6560 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:58:46.146063 6560 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:58:46.146116 6560 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:46.146155 6560 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:58:46.146180 6560 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:46.151963 6560 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37187
I20260926 03:58:46.152007 6688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37187 every 8 connection(s)
I20260926 03:58:46.152307 6560 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
I20260926 03:58:46.157975 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 6560
I20260926 03:58:46.158066 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-0/wal/instance
I20260926 03:58:46.158260 392 external_mini_cluster.cc:1428] Setting key 52dd9bc198ecadadd191dab19525c493
Sep 26 03:58:46 dist-test-slave-nhpw krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395126, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 03:58:46.161985 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:46.153028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38545 (local address 127.0.98.62:41421)
0926 03:58:46.153100 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:46.153117 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:46.153305 (+ 188us) server_negotiation.cc:407] Connection header received
0926 03:58:46.153816 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:46.153817 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:46.153829 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:46.153884 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:46.154793 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:46.155291 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:46.156293 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:46.156517 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:46.156612 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:46.158878 (+ 2266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:46.158880 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:46.158880 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:46.158887 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:46.160584 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:46.161030 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:46.161031 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:46.161032 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:46.161113 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:46.161390 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:46.161391 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:46.161391 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:46.161487 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:46.161538 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:46.161851 (+ 313us) server_negotiation.cc:299] Negotiation successful
0926 03:58:46.161922 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 03:58:46.162262 6691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:46.152987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41421 (local address 127.0.98.1:38545)
0926 03:58:46.153177 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:46.153221 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:46.153711 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:46.153950 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:46.153954 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:46.154511 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:46.154690 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:46.154695 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:46.155394 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:46.155395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:46.156201 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:46.156203 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:46.156621 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:46.156622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:46.156766 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:46.157289 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:46.157297 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:46.158795 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:58:46.160704 (+ 1909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:46.160709 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:46.160712 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:46.160944 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:46.161185 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:46.161186 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:46.161186 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:46.161326 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:46.161552 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:58:46.161557 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:58:46.161721 (+ 164us) client_negotiation.cc:769] Sending connection context
0926 03:58:46.161833 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 03:58:46.162002 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":96,"threads_started":1}
I20260926 03:58:46.162827 6689 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41421
I20260926 03:58:46.162942 6689 heartbeater.cc:464] Registering TS with master...
I20260926 03:58:46.163128 6689 heartbeater.cc:511] Master 127.0.98.62:41421 requested a full tablet report, sending...
I20260926 03:58:46.163662 6498 ts_manager.cc:194] Registered new tserver with Master: 226f08ac9eb34274ae439b88cea9050c (127.0.98.1:37187)
I20260926 03:58:46.164254 6498 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38545
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260926 03:58:46.183895 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41421
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:58:46.271660 6694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:58:46.271868 6694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:58:46.271895 6694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:58:46.273131 6694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:58:46.273173 6694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:58:46.273201 6694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 03:58:46.274693 6694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:58:46.274874 6694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:58:46.275058 6694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:58:46.280280 6700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:46.280315 6699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:46.280315 6702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:46.280444 6694 server_base.cc:1034] running on GCE node
I20260926 03:58:46.280769 6694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:58:46.281000 6694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:58:46.282166 6694 hybrid_clock.cc:648] HybridClock initialized: now 1790395126282152 us; error 28 us; skew 500 ppm
Sep 26 03:58:46 dist-test-slave-nhpw krb5kdc[6453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395126, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:58:46.284614 6694 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260926 03:58:46.285115 6694 webserver.cc:460] Webserver started at http://127.0.98.2:38885/ using document root <none> and password file <none>
I20260926 03:58:46.285344 6694 fs_manager.cc:359] Metadata directory not provided
I20260926 03:58:46.285413 6694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:58:46.285557 6694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:58:46.286566 6694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-1/data/instance:
uuid: "1e41eabd06d44a9ebab5ba2a76d5ea66"
format_stamp: "Formatted at 2026-09-26 03:58:46 on dist-test-slave-nhpw"
server_key: "455538b44176646187b9e6274d74fcc8"
server_key_iv: "e7d526d8b25608b1e9da2a5627b9776c"
server_key_version: "encryptionkey@0"
I20260926 03:58:46.286934 6694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-1/wal/instance:
uuid: "1e41eabd06d44a9ebab5ba2a76d5ea66"
format_stamp: "Formatted at 2026-09-26 03:58:46 on dist-test-slave-nhpw"
server_key: "455538b44176646187b9e6274d74fcc8"
server_key_iv: "e7d526d8b25608b1e9da2a5627b9776c"
server_key_version: "encryptionkey@0"
I20260926 03:58:46.288250 6694 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 03:58:46.289052 6709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:46.289225 6694 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:46.289319 6694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-1/wal
uuid: "1e41eabd06d44a9ebab5ba2a76d5ea66"
format_stamp: "Formatted at 2026-09-26 03:58:46 on dist-test-slave-nhpw"
server_key: "455538b44176646187b9e6274d74fcc8"
server_key_iv: "e7d526d8b25608b1e9da2a5627b9776c"
server_key_version: "encryptionkey@0"
I20260926 03:58:46.289407 6694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:58:46.310115 6694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:58:46.312096 6694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:58:46.312268 6694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:58:46.312494 6694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:58:46.312882 6694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:58:46.312943 6694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:46.312994 6694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:58:46.313030 6694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:46.318982 6694 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43115
I20260926 03:58:46.319016 6822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43115 every 8 connection(s)
I20260926 03:58:46.319353 6694 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
I20260926 03:58:46.320554 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 6694
I20260926 03:58:46.320688 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-1/wal/instance
I20260926 03:58:46.320876 392 external_mini_cluster.cc:1428] Setting key 6f7f129e6b5c4e4bad93cc0d675ed6e2
Sep 26 03:58:46 dist-test-slave-nhpw krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395126, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 03:58:46.329945 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:46.320071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50521 (local address 127.0.98.62:41421)
0926 03:58:46.320139 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:46.320142 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:46.320332 (+ 190us) server_negotiation.cc:407] Connection header received
0926 03:58:46.321159 (+ 827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:46.321160 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:46.321172 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:46.321254 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:46.322524 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:46.323069 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:46.324155 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:46.324380 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:46.324490 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:46.326838 (+ 2348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:46.326840 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:46.326841 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:46.326849 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:46.328334 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:46.328859 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:46.328860 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:46.328861 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:46.328925 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:46.329209 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:46.329210 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:46.329210 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:46.329329 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:46.329414 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:46.329755 (+ 341us) server_negotiation.cc:299] Negotiation successful
0926 03:58:46.329847 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 03:58:46.330161 6825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:46.320043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41421 (local address 127.0.98.2:50521)
0926 03:58:46.320210 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:46.320248 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:46.320948 (+ 700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:46.321326 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:46.321330 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:46.322144 (+ 814us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:46.322415 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:46.322421 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:46.323176 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:46.323178 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:46.324051 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:46.324053 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:46.324498 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:46.324499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:46.324628 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:46.325205 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:46.325214 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:46.326732 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:58:46.328441 (+ 1709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:46.328448 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:46.328450 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:46.328765 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:46.329012 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:46.329013 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:46.329013 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:46.329127 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:46.329404 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:58:46.329409 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:58:46.329585 (+ 176us) client_negotiation.cc:769] Sending connection context
0926 03:58:46.329718 (+ 133us) client_negotiation.cc:240] Negotiation successful
0926 03:58:46.329885 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":96,"threads_started":1}
I20260926 03:58:46.330709 6823 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41421
I20260926 03:58:46.330814 6823 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
I20260926 03:58:46.330999 6823 heartbeater.cc:511] Master 127.0.98.62:41421 requested a full tablet report, sending...
I20260926 03:58:46.331431 6498 ts_manager.cc:194] Registered new tserver with Master: 1e41eabd06d44a9ebab5ba2a76d5ea66 (127.0.98.2:43115)
I20260926 03:58:46.331796 6498 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50521
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260926 03:58:46.346992 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41421
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:58:46.426240 6828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:58:46.426481 6828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:58:46.426504 6828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:58:46.427610 6828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:58:46.427652 6828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:58:46.427677 6828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 03:58:46.429131 6828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:58:46.429356 6828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:58:46.429569 6828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:58:46.434474 6833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:46.434473 6836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:46.434568 6834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:46.434650 6828 server_base.cc:1034] running on GCE node
I20260926 03:58:46.434886 6828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:58:46.435073 6828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:58:46.436219 6828 hybrid_clock.cc:648] HybridClock initialized: now 1790395126436187 us; error 48 us; skew 500 ppm
Sep 26 03:58:46 dist-test-slave-nhpw krb5kdc[6453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395126, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:58:46.438650 6828 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260926 03:58:46.439064 6828 webserver.cc:460] Webserver started at http://127.0.98.3:38727/ using document root <none> and password file <none>
I20260926 03:58:46.439298 6828 fs_manager.cc:359] Metadata directory not provided
I20260926 03:58:46.439364 6828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:58:46.439515 6828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:58:46.440513 6828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-2/data/instance:
uuid: "241844d2ef2b4666bd8513fdc36a4efb"
format_stamp: "Formatted at 2026-09-26 03:58:46 on dist-test-slave-nhpw"
server_key: "79bc62c6f226e2e477d3e5764af86792"
server_key_iv: "9b66552daaf9270ed007ae462e87d421"
server_key_version: "encryptionkey@0"
I20260926 03:58:46.440920 6828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-2/wal/instance:
uuid: "241844d2ef2b4666bd8513fdc36a4efb"
format_stamp: "Formatted at 2026-09-26 03:58:46 on dist-test-slave-nhpw"
server_key: "79bc62c6f226e2e477d3e5764af86792"
server_key_iv: "9b66552daaf9270ed007ae462e87d421"
server_key_version: "encryptionkey@0"
I20260926 03:58:46.442497 6828 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 03:58:46.443301 6843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:46.443502 6828 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 03:58:46.443578 6828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-2/wal
uuid: "241844d2ef2b4666bd8513fdc36a4efb"
format_stamp: "Formatted at 2026-09-26 03:58:46 on dist-test-slave-nhpw"
server_key: "79bc62c6f226e2e477d3e5764af86792"
server_key_iv: "9b66552daaf9270ed007ae462e87d421"
server_key_version: "encryptionkey@0"
I20260926 03:58:46.443629 6828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:58:46.458027 6828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:58:46.459865 6828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:58:46.459977 6828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:58:46.460155 6828 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:58:46.460386 6828 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:58:46.460414 6828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:46.460435 6828 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:58:46.460448 6828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:46.466176 6828 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43813
I20260926 03:58:46.466224 6956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43813 every 8 connection(s)
I20260926 03:58:46.466527 6828 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
I20260926 03:58:46.471280 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 6828
I20260926 03:58:46.471359 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-2/wal/instance
I20260926 03:58:46.471534 392 external_mini_cluster.cc:1428] Setting key 539648ecd80cc8ce5df9cf5c60d24db8
Sep 26 03:58:46 dist-test-slave-nhpw krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395126, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 03:58:46.475639 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:46.467133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56543 (local address 127.0.98.62:41421)
0926 03:58:46.467196 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:46.467198 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:46.467422 (+ 224us) server_negotiation.cc:407] Connection header received
0926 03:58:46.467866 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:46.467867 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:46.467877 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:46.467922 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:46.468889 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:46.469378 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:46.470284 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:46.470456 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:46.470508 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:46.472792 (+ 2284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:46.472793 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:46.472793 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:46.472799 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:46.474245 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:46.474677 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:46.474678 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:46.474679 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:46.474743 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:46.474987 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:46.474987 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:46.474988 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:46.475148 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:46.475217 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:46.475521 (+ 304us) server_negotiation.cc:299] Negotiation successful
0926 03:58:46.475578 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 03:58:46.475901 6959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:46.467157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41421 (local address 127.0.98.3:56543)
0926 03:58:46.467307 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:46.467339 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:46.467779 (+ 440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:46.468023 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:46.468026 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:46.468555 (+ 529us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:46.468815 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:46.468820 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:46.469452 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:46.469453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:46.470213 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:46.470215 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:46.470550 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:46.470551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:46.470656 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:46.471057 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:46.471065 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:46.472708 (+ 1643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:58:46.474334 (+ 1626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:46.474339 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:46.474341 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:46.474578 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:46.474808 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:46.474808 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:46.474809 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:46.474924 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:46.475211 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:58:46.475215 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:58:46.475395 (+ 180us) client_negotiation.cc:769] Sending connection context
0926 03:58:46.475506 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 03:58:46.475628 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":67,"threads_started":1}
I20260926 03:58:46.476440 6957 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41421
I20260926 03:58:46.476545 6957 heartbeater.cc:464] Registering TS with master...
I20260926 03:58:46.476768 6957 heartbeater.cc:511] Master 127.0.98.62:41421 requested a full tablet report, sending...
I20260926 03:58:46.477137 6498 ts_manager.cc:194] Registered new tserver with Master: 241844d2ef2b4666bd8513fdc36a4efb (127.0.98.3:43813)
I20260926 03:58:46.477496 6498 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56543
I20260926 03:58:46.483134 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260926 03:58:46.507267 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:35627
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41421,127.0.98.61:35627
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:58:46.585759 6962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:58:46.585948 6962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:58:46.585973 6962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:58:46.587092 6962 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:58:46.587132 6962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:58:46.587146 6962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:58:46.587188 6962 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:58:46.587216 6962 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:58:46.588610 6962 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41421,127.0.98.61:35627
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:35627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
W20260926 03:58:46.588747 6962 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:41421,127.0.98.61:35627'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260926 03:58:46.588886 6962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:58:46.589098 6962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:58:46.593776 6970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:46.593809 6968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:46.593840 6967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:46.593892 6962 server_base.cc:1034] running on GCE node
I20260926 03:58:46.594343 6962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:58:46.594596 6962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:58:46.595736 6962 hybrid_clock.cc:648] HybridClock initialized: now 1790395126595715 us; error 60 us; skew 500 ppm
Sep 26 03:58:46 dist-test-slave-nhpw krb5kdc[6453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395126, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:58:46.598250 6962 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260926 03:58:46.598703 6962 webserver.cc:460] Webserver started at http://127.0.98.61:41629/ using document root <none> and password file <none>
I20260926 03:58:46.598925 6962 fs_manager.cc:359] Metadata directory not provided
I20260926 03:58:46.598992 6962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:58:46.599144 6962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:58:46.600162 6962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/master-1/data/instance:
uuid: "3f6ed0be592b4c6da0be1ae5feb24ff1"
format_stamp: "Formatted at 2026-09-26 03:58:46 on dist-test-slave-nhpw"
server_key: "067ec03b1930d0466c94977c37e7ce3d"
server_key_iv: "19c8ef35add59c12b4ea79d5395b87f3"
server_key_version: "encryptionkey@0"
I20260926 03:58:46.600518 6962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/master-1/wal/instance:
uuid: "3f6ed0be592b4c6da0be1ae5feb24ff1"
format_stamp: "Formatted at 2026-09-26 03:58:46 on dist-test-slave-nhpw"
server_key: "067ec03b1930d0466c94977c37e7ce3d"
server_key_iv: "19c8ef35add59c12b4ea79d5395b87f3"
server_key_version: "encryptionkey@0"
I20260926 03:58:46.601840 6962 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 03:58:46.602546 6977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:46.602720 6962 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 03:58:46.602806 6962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/master-1/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/master-1/wal
uuid: "3f6ed0be592b4c6da0be1ae5feb24ff1"
format_stamp: "Formatted at 2026-09-26 03:58:46 on dist-test-slave-nhpw"
server_key: "067ec03b1930d0466c94977c37e7ce3d"
server_key_iv: "19c8ef35add59c12b4ea79d5395b87f3"
server_key_version: "encryptionkey@0"
I20260926 03:58:46.602886 6962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:58:46.627880 6962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:58:46.630034 6962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:58:46.630196 6962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:58:46.634346 6962 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:35627
I20260926 03:58:46.634379 7029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:35627 every 8 connection(s)
I20260926 03:58:46.634721 6962 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/master-1/data/info.pb
I20260926 03:58:46.635697 7030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:58:46.637351 7030 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41421 } has no permanent_uuid. Determining permanent_uuid...
I20260926 03:58:46.641626 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 6962
Sep 26 03:58:46 dist-test-slave-nhpw krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395126, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 03:58:46.646739 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:46.638056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47540 (local address 127.0.98.62:41421)
0926 03:58:46.638161 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:46.638163 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:46.638318 (+ 155us) server_negotiation.cc:407] Connection header received
0926 03:58:46.638818 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:46.638820 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:46.638834 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:46.638889 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:46.639477 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:46.639968 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:46.641034 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:46.641225 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:46.641308 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:46.643844 (+ 2536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:46.643845 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:46.643846 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:46.643854 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:46.645281 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:46.645700 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:46.645701 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:46.645702 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:46.645761 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:46.646010 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:46.646011 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:46.646012 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:46.646212 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:46.646295 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:46.646575 (+ 280us) server_negotiation.cc:299] Negotiation successful
0926 03:58:46.646651 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260926 03:58:46.646935 7032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:46.638022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41421 (local address 127.0.0.1:47540)
0926 03:58:46.638200 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:46.638239 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:46.638703 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:46.638970 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:46.638973 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:46.639204 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:46.639381 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:46.639385 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:46.640060 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:46.640061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:46.640939 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:46.640941 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:46.641317 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:46.641318 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:46.641431 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:46.642068 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:46.642077 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:46.643755 (+ 1678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:58:46.645382 (+ 1627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:46.645389 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:46.645391 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:46.645610 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:46.645826 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:46.645827 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:46.645828 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:46.645941 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:46.646297 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:58:46.646303 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:58:46.646445 (+ 142us) client_negotiation.cc:769] Sending connection context
0926 03:58:46.646552 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 03:58:46.646700 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":80,"threads_started":1}
Sep 26 03:58:46 dist-test-slave-nhpw krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260926 03:58:46.647478 7030 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35627 } has no permanent_uuid. Determining permanent_uuid...
I20260926 03:58:46.651230 7034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:46.642181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47000 (local address 127.0.98.61:35627)
0926 03:58:46.642342 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:46.642347 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:46.642370 (+ 23us) server_negotiation.cc:407] Connection header received
0926 03:58:46.642393 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:46.642396 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:46.642417 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:46.642512 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:46.643176 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:46.643959 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:46.644821 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:46.645135 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:46.645220 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:46.647992 (+ 2772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:46.647997 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:46.647997 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:46.648003 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:46.649897 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:46.650276 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:46.650279 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:46.650279 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:46.650333 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:46.650543 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:46.650544 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:46.650544 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:46.650723 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:46.650874 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:46.650974 (+ 100us) server_negotiation.cc:299] Negotiation successful
0926 03:58:46.651101 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":76,"threads_started":1}
Sep 26 03:58:46 dist-test-slave-nhpw krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395126, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260926 03:58:46.655195 7032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:46.647755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35627 (local address 127.0.0.1:47006)
0926 03:58:46.647813 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:46.647826 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:46.647877 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:46.648063 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:46.648064 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:46.648249 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:46.648306 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:46.648308 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:46.649184 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:46.649185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:46.650063 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:46.650065 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:46.650545 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:46.650546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:46.650629 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:46.651063 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:46.651068 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:46.652359 (+ 1291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:58:46.654312 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:46.654313 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:46.654313 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:46.654467 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:46.654638 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:46.654639 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:46.654639 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:46.654686 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:46.654918 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:58:46.654920 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:58:46.654959 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 03:58:46.655028 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 03:58:46.655097 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 03:58:46.655298 7035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:46.647797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47006 (local address 127.0.98.61:35627)
0926 03:58:46.647940 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:46.647942 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:46.647949 (+ 7us) server_negotiation.cc:407] Connection header received
0926 03:58:46.647967 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:46.647967 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:46.647983 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:46.648038 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:46.648383 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:46.649098 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:46.650142 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:46.650459 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:46.650528 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:46.652527 (+ 1999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:46.652529 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:46.652529 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:46.652535 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:46.654243 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:46.654534 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:46.654535 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:46.654535 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:46.654582 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:46.654738 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:46.654738 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:46.654738 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:46.654823 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:46.654869 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:46.655078 (+ 209us) server_negotiation.cc:299] Negotiation successful
0926 03:58:46.655139 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":125,"thread_start_us":102,"threads_started":1}
I20260926 03:58:46.656473 7030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f6ed0be592b4c6da0be1ae5feb24ff1: Bootstrap starting.
I20260926 03:58:46.657140 7030 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f6ed0be592b4c6da0be1ae5feb24ff1: Neither blocks nor log segments found. Creating new log.
I20260926 03:58:46.657413 7030 log.cc:824] T 00000000000000000000000000000000 P 3f6ed0be592b4c6da0be1ae5feb24ff1: Log is configured to *not* fsync() on all Append() calls
I20260926 03:58:46.658066 7030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f6ed0be592b4c6da0be1ae5feb24ff1: No bootstrap required, opened a new log
I20260926 03:58:46.659379 7030 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f6ed0be592b4c6da0be1ae5feb24ff1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aeea5ec3a114f8c94fd2d2012fe312b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41421 } } peers { permanent_uuid: "3f6ed0be592b4c6da0be1ae5feb24ff1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35627 } }
I20260926 03:58:46.659523 7030 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f6ed0be592b4c6da0be1ae5feb24ff1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:58:46.659592 7030 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f6ed0be592b4c6da0be1ae5feb24ff1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f6ed0be592b4c6da0be1ae5feb24ff1, State: Initialized, Role: FOLLOWER
I20260926 03:58:46.659742 7030 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f6ed0be592b4c6da0be1ae5feb24ff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aeea5ec3a114f8c94fd2d2012fe312b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41421 } } peers { permanent_uuid: "3f6ed0be592b4c6da0be1ae5feb24ff1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35627 } }
I20260926 03:58:46.660025 7037 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f6ed0be592b4c6da0be1ae5feb24ff1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aeea5ec3a114f8c94fd2d2012fe312b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41421 } } peers { permanent_uuid: "3f6ed0be592b4c6da0be1ae5feb24ff1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35627 } } }
I20260926 03:58:46.660089 7030 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f6ed0be592b4c6da0be1ae5feb24ff1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:58:46.660132 7037 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f6ed0be592b4c6da0be1ae5feb24ff1 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 03:58:46.661753 7048 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3f6ed0be592b4c6da0be1ae5feb24ff1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 03:58:46.661809 7048 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 03:58:46.663251 6962 master_runner.cc:428] Detected that this master 3f6ed0be592b4c6da0be1ae5feb24ff1 is joining an existing cluster
I20260926 03:58:46.663318 6962 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:35627
I20260926 03:58:46.671075 7035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:46.664456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47018 (local address 127.0.98.61:35627)
0926 03:58:46.664492 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:46.664494 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:46.664612 (+ 118us) server_negotiation.cc:407] Connection header received
0926 03:58:46.664676 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:46.664677 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:46.664687 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:46.664715 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:46.665194 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:46.665920 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:46.667011 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:46.667297 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:46.667356 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:46.668337 (+ 981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:46.668338 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:46.668339 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:46.668344 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:46.669926 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:46.670285 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:46.670287 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:46.670287 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:46.670335 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:46.670534 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:46.670535 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:46.670536 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:46.670646 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:46.670718 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:46.670868 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 03:58:46.670952 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":18}
I20260926 03:58:46.671408 7054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:46.664379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35627 (local address 127.0.0.1:47018)
0926 03:58:46.664537 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:46.664547 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:46.664606 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:46.664772 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:46.664773 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:46.665038 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:46.665104 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:46.665106 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:46.666019 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:46.666020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:46.666928 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:46.666931 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:46.667403 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:46.667403 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:46.667440 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:46.667860 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:46.667865 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:46.668247 (+ 382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:58:46.670027 (+ 1780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:46.670028 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:46.670029 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:46.670200 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:46.670407 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:46.670408 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:46.670409 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:46.670464 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:46.670709 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:58:46.670710 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:58:46.670746 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 03:58:46.670847 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 03:58:46.671002 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":96,"thread_start_us":67,"threads_started":1}
I20260926 03:58:46.671437 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:46.664244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47548 (local address 127.0.98.62:41421)
0926 03:58:46.664285 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:46.664287 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:46.664657 (+ 370us) server_negotiation.cc:407] Connection header received
0926 03:58:46.664722 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:46.664722 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:46.664733 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:46.664789 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:46.665158 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:46.665921 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:46.666762 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:46.666990 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:46.667062 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:46.668343 (+ 1281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:46.668344 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:46.668345 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:46.668353 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:46.670220 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:46.670626 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:46.670627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:46.670628 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:46.670680 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:46.670872 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:46.670872 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:46.670873 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:46.670986 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:46.671027 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:46.671270 (+ 243us) server_negotiation.cc:299] Negotiation successful
0926 03:58:46.671343 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 03:58:46.671432 7055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:46.664193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41421 (local address 127.0.0.1:47548)
0926 03:58:46.664530 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:46.664539 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:46.664629 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:46.664794 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:46.664795 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:46.665016 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:46.665080 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:46.665082 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:46.666013 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:46.666014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:46.666671 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:46.666673 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:46.667128 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:46.667129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:46.667190 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:46.667689 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:46.667693 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:46.668247 (+ 554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:58:46.670339 (+ 2092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:46.670340 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:46.670341 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:46.670542 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:46.670751 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:46.670751 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:46.670751 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:46.670802 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:46.671074 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:58:46.671075 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:58:46.671137 (+ 62us) client_negotiation.cc:769] Sending connection context
0926 03:58:46.671231 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 03:58:46.671323 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"mutex_wait_us":57,"thread_start_us":71,"threads_started":1}
Sep 26 03:58:46 dist-test-slave-nhpw krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395125, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260926 03:58:46.681401 7035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:46.673839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47030 (local address 127.0.98.61:35627)
0926 03:58:46.673879 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:46.673881 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:46.674022 (+ 141us) server_negotiation.cc:407] Connection header received
0926 03:58:46.674080 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:46.674080 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:46.674095 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:46.674161 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:46.674499 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:46.675211 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:46.676005 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:46.676305 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:46.676375 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:46.678557 (+ 2182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:46.678559 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:46.678559 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:46.678566 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:46.680187 (+ 1621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:46.680583 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:46.680584 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:46.680584 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:46.680661 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:46.680897 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:46.680898 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:46.680898 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:46.681018 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:46.681090 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:46.681283 (+ 193us) server_negotiation.cc:299] Negotiation successful
0926 03:58:46.681341 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 03:58:46.681542 7056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:46.673796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35627 (local address 127.0.0.1:47030)
0926 03:58:46.673910 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:46.673951 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:46.674021 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:46.674164 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:46.674166 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:46.674392 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:46.674434 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:46.674435 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:46.675315 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:46.675316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:46.675910 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:46.675912 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:46.676380 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:46.676381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:46.676439 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:46.676935 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:46.676940 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:46.678459 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:58:46.680271 (+ 1812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:46.680278 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:46.680280 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:46.680493 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:46.680742 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:46.680743 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:46.680744 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:46.680808 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:46.681102 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:58:46.681109 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:58:46.681164 (+ 55us) client_negotiation.cc:769] Sending connection context
0926 03:58:46.681256 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 03:58:46.681375 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":67,"threads_started":1}
I20260926 03:58:46.682143 6498 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:35627: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "3f6ed0be592b4c6da0be1ae5feb24ff1"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 35627
}
attrs {
promote: true
}
}
dest_uuid: "0aeea5ec3a114f8c94fd2d2012fe312b"
cas_config_opid_index: -1
I20260926 03:58:46.682448 6498 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0aeea5ec3a114f8c94fd2d2012fe312b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0aeea5ec3a114f8c94fd2d2012fe312b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41421 } } peers { permanent_uuid: "3f6ed0be592b4c6da0be1ae5feb24ff1" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35627 } attrs { promote: true } }
I20260926 03:58:46.682919 7058 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0aeea5ec3a114f8c94fd2d2012fe312b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 3f6ed0be592b4c6da0be1ae5feb24ff1 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0aeea5ec3a114f8c94fd2d2012fe312b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41421 } } peers { permanent_uuid: "3f6ed0be592b4c6da0be1ae5feb24ff1" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35627 } attrs { promote: true } } }
I20260926 03:58:46.683212 7004 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f6ed0be592b4c6da0be1ae5feb24ff1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:58:46.683966 7004 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3f6ed0be592b4c6da0be1ae5feb24ff1 [term 1 FOLLOWER]: Refusing update from remote peer 0aeea5ec3a114f8c94fd2d2012fe312b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260926 03:58:46.684302 7057 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0aeea5ec3a114f8c94fd2d2012fe312b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f6ed0be592b4c6da0be1ae5feb24ff1" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35627 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:58:46.685061 7058 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:35627
I20260926 03:58:46.685148 7058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0aeea5ec3a114f8c94fd2d2012fe312b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0aeea5ec3a114f8c94fd2d2012fe312b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0aeea5ec3a114f8c94fd2d2012fe312b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41421 } } peers { permanent_uuid: "3f6ed0be592b4c6da0be1ae5feb24ff1" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35627 } attrs { promote: true } } }
I20260926 03:58:46.685205 7058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0aeea5ec3a114f8c94fd2d2012fe312b [sys.catalog]: This master's current role is: LEADER
I20260926 03:58:46.685294 6962 master.cc:537] Master@127.0.98.61:35627 shutting down...
I20260926 03:58:46.686367 7004 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3f6ed0be592b4c6da0be1ae5feb24ff1 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 3f6ed0be592b4c6da0be1ae5feb24ff1 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0aeea5ec3a114f8c94fd2d2012fe312b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41421 } } peers { permanent_uuid: "3f6ed0be592b4c6da0be1ae5feb24ff1" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35627 } attrs { promote: true } } }
I20260926 03:58:46.687304 7037 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f6ed0be592b4c6da0be1ae5feb24ff1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0aeea5ec3a114f8c94fd2d2012fe312b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0aeea5ec3a114f8c94fd2d2012fe312b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41421 } } peers { permanent_uuid: "3f6ed0be592b4c6da0be1ae5feb24ff1" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35627 } attrs { promote: true } } }
I20260926 03:58:46.687382 7037 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f6ed0be592b4c6da0be1ae5feb24ff1 [sys.catalog]: This master's current role is: LEARNER
I20260926 03:58:46.687484 7066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f6ed0be592b4c6da0be1ae5feb24ff1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0aeea5ec3a114f8c94fd2d2012fe312b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0aeea5ec3a114f8c94fd2d2012fe312b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41421 } } peers { permanent_uuid: "3f6ed0be592b4c6da0be1ae5feb24ff1" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35627 } attrs { promote: true } } }
I20260926 03:58:46.687587 7066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f6ed0be592b4c6da0be1ae5feb24ff1 [sys.catalog]: This master's current role is: LEARNER
I20260926 03:58:46.687950 7062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f6ed0be592b4c6da0be1ae5feb24ff1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0aeea5ec3a114f8c94fd2d2012fe312b. Latest consensus state: current_term: 1 leader_uuid: "0aeea5ec3a114f8c94fd2d2012fe312b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0aeea5ec3a114f8c94fd2d2012fe312b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41421 } } peers { permanent_uuid: "3f6ed0be592b4c6da0be1ae5feb24ff1" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35627 } attrs { promote: true } } }
I20260926 03:58:46.688069 7062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f6ed0be592b4c6da0be1ae5feb24ff1 [sys.catalog]: This master's current role is: LEARNER
I20260926 03:58:46.689421 6962 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 3f6ed0be592b4c6da0be1ae5feb24ff1 [term 1 LEARNER]: Raft consensus shutting down.
I20260926 03:58:46.689540 6962 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 3f6ed0be592b4c6da0be1ae5feb24ff1 [term 1 LEARNER]: Raft consensus is shut down!
I20260926 03:58:46.689581 6962 tablet_replica.cc:330] T 00000000000000000000000000000000 P 3f6ed0be592b4c6da0be1ae5feb24ff1: stopping tablet replica
I20260926 03:58:46.702574 6962 master.cc:552] Master@127.0.98.61:35627 shutdown complete.
I20260926 03:58:46.702658 6962 master_runner.cc:305] Clearing existing system tablet
I20260926 03:58:46.702791 6962 fs_manager.cc:359] Metadata directory not provided
I20260926 03:58:46.702848 6962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:58:46.703469 6962 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.001s
I20260926 03:58:46.703960 7068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:46.704110 6962 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:58:46.704178 6962 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:46.704203 6962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/master-1/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/master-1/wal
uuid: "3f6ed0be592b4c6da0be1ae5feb24ff1"
format_stamp: "Formatted at 2026-09-26 03:58:46 on dist-test-slave-nhpw"
server_key: "067ec03b1930d0466c94977c37e7ce3d"
server_key_iv: "19c8ef35add59c12b4ea79d5395b87f3"
server_key_version: "encryptionkey@0"
I20260926 03:58:46.704979 6962 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 3f6ed0be592b4c6da0be1ae5feb24ff1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:58:46.706696 6962 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 3f6ed0be592b4c6da0be1ae5feb24ff1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 03:58:46.706766 6962 log.cc:1196] T 00000000000000000000000000000000 P 3f6ed0be592b4c6da0be1ae5feb24ff1: Deleting WAL directory at /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260926 03:58:46.707003 6962 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 3f6ed0be592b4c6da0be1ae5feb24ff1: Deleting consensus metadata
I20260926 03:58:46.707129 6962 master_runner.cc:315] Copying system tablet from 127.0.98.62:41421
I20260926 03:58:46.707700 6962 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 3f6ed0be592b4c6da0be1ae5feb24ff1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:41421
I20260926 03:58:46.714561 7074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:46.707968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41421 (local address 127.0.0.1:47552)
0926 03:58:46.708103 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:46.708114 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:46.708179 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:46.708363 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:46.708365 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:46.708536 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:46.708586 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:46.708588 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:46.709685 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:46.709686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:46.710492 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:46.710496 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:46.710976 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:46.710978 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:46.711036 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:46.711584 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:46.711588 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:46.711867 (+ 279us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:58:46.713618 (+ 1751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:46.713619 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:46.713619 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:46.713776 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:46.713987 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:46.713988 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:46.713988 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:46.714035 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:46.714317 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:58:46.714318 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:58:46.714353 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 03:58:46.714403 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 03:58:46.714477 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":96,"threads_started":1}
I20260926 03:58:46.714594 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:46.708040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47552 (local address 127.0.98.62:41421)
0926 03:58:46.708100 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:46.708103 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:46.708180 (+ 77us) server_negotiation.cc:407] Connection header received
0926 03:58:46.708252 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:46.708253 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:46.708263 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:46.708324 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:46.708752 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:46.709591 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:46.710593 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:46.710906 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:46.710977 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:46.711954 (+ 977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:46.711956 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:46.711956 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:46.711963 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:46.713536 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:46.713848 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:46.713849 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:46.713850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:46.713911 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:46.714125 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:46.714125 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:46.714126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:46.714255 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:46.714324 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:46.714430 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 03:58:46.714501 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 03:58:46.714897 6518 tablet_copy_service.cc:140] P 0aeea5ec3a114f8c94fd2d2012fe312b: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 3f6ed0be592b4c6da0be1ae5feb24ff1 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:47552)
I20260926 03:58:46.715001 6518 tablet_copy_service.cc:161] P 0aeea5ec3a114f8c94fd2d2012fe312b: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 3f6ed0be592b4c6da0be1ae5feb24ff1 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:47552: session id = 3f6ed0be592b4c6da0be1ae5feb24ff1-00000000000000000000000000000000
I20260926 03:58:46.715624 6518 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 0aeea5ec3a114f8c94fd2d2012fe312b: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 03:58:46.716267 6962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 03:58:46.717482 6962 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 3f6ed0be592b4c6da0be1ae5feb24ff1: tablet copy: Starting download of 0 data blocks...
I20260926 03:58:46.717634 6962 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 3f6ed0be592b4c6da0be1ae5feb24ff1: tablet copy: Starting download of 1 WAL segments...
I20260926 03:58:46.718472 6962 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 3f6ed0be592b4c6da0be1ae5feb24ff1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 03:58:46.719332 6518 tablet_copy_service.cc:342] P 0aeea5ec3a114f8c94fd2d2012fe312b: Request end of tablet copy session 3f6ed0be592b4c6da0be1ae5feb24ff1-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:47552
I20260926 03:58:46.719403 6518 tablet_copy_service.cc:433] P 0aeea5ec3a114f8c94fd2d2012fe312b: ending tablet copy session 3f6ed0be592b4c6da0be1ae5feb24ff1-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 3f6ed0be592b4c6da0be1ae5feb24ff1
I20260926 03:58:46.721913 6962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:58:46.723126 7075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:46.723186 7076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:46.723301 6962 server_base.cc:1034] running on GCE node
W20260926 03:58:46.724763 7078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:46.724892 6962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:58:46.725032 6962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:58:46.726154 6962 hybrid_clock.cc:648] HybridClock initialized: now 1790395126726135 us; error 38 us; skew 500 ppm
Sep 26 03:58:46 dist-test-slave-nhpw krb5kdc[6453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395126, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:58:46.727509 6962 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260926 03:58:46.727811 6962 webserver.cc:460] Webserver started at http://127.0.98.61:43661/ using document root <none> and password file <none>
I20260926 03:58:46.727959 6962 fs_manager.cc:359] Metadata directory not provided
I20260926 03:58:46.727998 6962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:58:46.728597 6962 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260926 03:58:46.729156 7085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:46.729271 6962 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:46.729317 6962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/master-1/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/master-1/wal
uuid: "3f6ed0be592b4c6da0be1ae5feb24ff1"
format_stamp: "Formatted at 2026-09-26 03:58:46 on dist-test-slave-nhpw"
server_key: "067ec03b1930d0466c94977c37e7ce3d"
server_key_iv: "19c8ef35add59c12b4ea79d5395b87f3"
server_key_version: "encryptionkey@0"
I20260926 03:58:46.729370 6962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:58:46.733848 6962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:58:46.735298 6962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:58:46.737783 6962 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:35627
I20260926 03:58:46.738171 6962 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/master-1/data/info.pb
I20260926 03:58:46.738190 7137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:35627 every 8 connection(s)
I20260926 03:58:46.739007 7138 sys_catalog.cc:263] Verifying existing consensus state
I20260926 03:58:46.739502 7138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f6ed0be592b4c6da0be1ae5feb24ff1: Bootstrap starting.
I20260926 03:58:46.741776 7138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f6ed0be592b4c6da0be1ae5feb24ff1: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 03:58:46.742017 7138 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f6ed0be592b4c6da0be1ae5feb24ff1: Bootstrap complete.
I20260926 03:58:46.742211 7138 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f6ed0be592b4c6da0be1ae5feb24ff1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0aeea5ec3a114f8c94fd2d2012fe312b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41421 } } peers { permanent_uuid: "3f6ed0be592b4c6da0be1ae5feb24ff1" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35627 } attrs { promote: true } }
I20260926 03:58:46.742316 7138 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f6ed0be592b4c6da0be1ae5feb24ff1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3f6ed0be592b4c6da0be1ae5feb24ff1, State: Initialized, Role: LEARNER
I20260926 03:58:46.742354 7138 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f6ed0be592b4c6da0be1ae5feb24ff1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0aeea5ec3a114f8c94fd2d2012fe312b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41421 } } peers { permanent_uuid: "3f6ed0be592b4c6da0be1ae5feb24ff1" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35627 } attrs { promote: true } }
I20260926 03:58:46.742537 7138 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f6ed0be592b4c6da0be1ae5feb24ff1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:58:46.742537 7141 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f6ed0be592b4c6da0be1ae5feb24ff1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "0aeea5ec3a114f8c94fd2d2012fe312b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41421 } } peers { permanent_uuid: "3f6ed0be592b4c6da0be1ae5feb24ff1" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35627 } attrs { promote: true } } }
I20260926 03:58:46.742683 7141 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f6ed0be592b4c6da0be1ae5feb24ff1 [sys.catalog]: This master's current role is: LEARNER
I20260926 03:58:46.744061 7152 catalog_manager.cc:1259] Loaded cluster ID: a989010329ac4d63b6f8b327d43ccef5
I20260926 03:58:46.744282 7152 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3f6ed0be592b4c6da0be1ae5feb24ff1: loading cluster ID for follower catalog manager: success
I20260926 03:58:46.745106 7152 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3f6ed0be592b4c6da0be1ae5feb24ff1: acquiring CA information for follower catalog manager: success
I20260926 03:58:46.745379 7152 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3f6ed0be592b4c6da0be1ae5feb24ff1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 03:58:46.760192 7153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:46.752981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47042 (local address 127.0.98.61:35627)
0926 03:58:46.753109 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:46.753111 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:46.753121 (+ 10us) server_negotiation.cc:407] Connection header received
0926 03:58:46.753152 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:46.753153 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:46.753161 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:46.753223 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:46.753915 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:46.754560 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:46.755603 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:46.755808 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:46.755876 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:46.757340 (+ 1464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:46.757341 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:46.757341 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:46.757347 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:46.758845 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:46.759247 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:46.759248 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:46.759248 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:46.759299 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:46.759500 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:46.759501 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:46.759501 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:46.759591 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:46.759667 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:46.759854 (+ 187us) server_negotiation.cc:299] Negotiation successful
0926 03:58:46.759995 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":81,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260926 03:58:46.784809 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:34077
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41421,127.0.98.61:35627,127.0.98.60:34077
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:58:46.873296 7156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:58:46.873499 7156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:58:46.873520 7156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:58:46.874675 7156 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:58:46.874717 7156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:58:46.874732 7156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:58:46.874744 7156 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:58:46.874756 7156 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:58:46.876247 7156 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41421,127.0.98.61:35627,127.0.98.60:34077
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:34077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:58:46.876502 7156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:58:46.876767 7156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 03:58:46.881696 7156 server_base.cc:1034] running on GCE node
W20260926 03:58:46.881656 7161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:46.881673 7162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:46.881678 7164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:46.882185 7156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:58:46.882483 7156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:58:46.883646 7156 hybrid_clock.cc:648] HybridClock initialized: now 1790395126883628 us; error 31 us; skew 500 ppm
Sep 26 03:58:46 dist-test-slave-nhpw krb5kdc[6453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395126, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:58:46.886289 7156 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260926 03:58:46.886749 7156 webserver.cc:460] Webserver started at http://127.0.98.60:45941/ using document root <none> and password file <none>
I20260926 03:58:46.886987 7156 fs_manager.cc:359] Metadata directory not provided
I20260926 03:58:46.887033 7156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:58:46.887187 7156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:58:46.888286 7156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/master-2/data/instance:
uuid: "e84368bd28c54795852888b3d9163a75"
format_stamp: "Formatted at 2026-09-26 03:58:46 on dist-test-slave-nhpw"
server_key: "c25e61eb5e7a6127e8e669dca1a3d9a1"
server_key_iv: "042d7203e66df8902fdfe5871e78449c"
server_key_version: "encryptionkey@0"
I20260926 03:58:46.888641 7156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/master-2/wal/instance:
uuid: "e84368bd28c54795852888b3d9163a75"
format_stamp: "Formatted at 2026-09-26 03:58:46 on dist-test-slave-nhpw"
server_key: "c25e61eb5e7a6127e8e669dca1a3d9a1"
server_key_iv: "042d7203e66df8902fdfe5871e78449c"
server_key_version: "encryptionkey@0"
I20260926 03:58:46.889984 7156 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 03:58:46.890769 7171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:46.890978 7156 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 03:58:46.891077 7156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/master-2/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/master-2/wal
uuid: "e84368bd28c54795852888b3d9163a75"
format_stamp: "Formatted at 2026-09-26 03:58:46 on dist-test-slave-nhpw"
server_key: "c25e61eb5e7a6127e8e669dca1a3d9a1"
server_key_iv: "042d7203e66df8902fdfe5871e78449c"
server_key_version: "encryptionkey@0"
I20260926 03:58:46.891178 7156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:58:46.901835 7156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:58:46.904062 7156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:58:46.904215 7156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:58:46.907859 7156 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:34077
I20260926 03:58:46.907909 7223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:34077 every 8 connection(s)
I20260926 03:58:46.908254 7156 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/master-2/data/info.pb
I20260926 03:58:46.909185 7224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:58:46.909691 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 7156
I20260926 03:58:46.911813 7224 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41421 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:58:46 dist-test-slave-nhpw krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395126, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 26 03:58:46 dist-test-slave-nhpw krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395125, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260926 03:58:46.920306 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:46.912591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47562 (local address 127.0.98.62:41421)
0926 03:58:46.912669 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:46.912671 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:46.912742 (+ 71us) server_negotiation.cc:407] Connection header received
0926 03:58:46.912808 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:46.912809 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:46.912817 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:46.912865 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:46.913377 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:46.913843 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:46.914793 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:46.914970 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:46.915048 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:46.917080 (+ 2032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:46.917082 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:46.917082 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:46.917090 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:46.918666 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:46.919353 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:46.919354 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:46.919354 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:46.919404 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:46.919673 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:46.919674 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:46.919674 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:46.919793 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:46.919869 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:46.920136 (+ 267us) server_negotiation.cc:299] Negotiation successful
0926 03:58:46.920212 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 03:58:46.920502 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:46.912460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41421 (local address 127.0.0.1:47562)
0926 03:58:46.912616 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:46.912677 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:46.912747 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:46.912900 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:46.912903 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:46.913240 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:46.913307 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:46.913308 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:46.913939 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:46.913940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:46.914701 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:46.914704 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:46.915056 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:46.915057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:46.915138 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:46.915583 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:46.915590 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:46.916986 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:58:46.918876 (+ 1890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:46.918880 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:46.918882 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:46.919271 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:46.919481 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:46.919482 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:46.919482 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:46.919599 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:46.919860 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:58:46.919865 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:58:46.920005 (+ 140us) client_negotiation.cc:769] Sending connection context
0926 03:58:46.920115 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 03:58:46.920267 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":75,"threads_started":1}
I20260926 03:58:46.921036 7224 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35627 } has no permanent_uuid. Determining permanent_uuid...
I20260926 03:58:46.921113 7226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:46.910327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45368 (local address 127.0.98.60:34077)
0926 03:58:46.910480 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:46.910489 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:46.910522 (+ 33us) server_negotiation.cc:407] Connection header received
0926 03:58:46.911084 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:46.911093 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:46.911156 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:46.911341 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:46.911944 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:46.913299 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:46.914113 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:46.914499 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:46.914578 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:46.917494 (+ 2916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:46.917500 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:46.917501 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:46.917507 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:46.919386 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:46.919744 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:46.919747 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:46.919747 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:46.919808 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:46.920027 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:46.920028 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:46.920028 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:46.920163 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:46.920311 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:46.920918 (+ 607us) server_negotiation.cc:299] Negotiation successful
0926 03:58:46.920994 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":82,"threads_started":1}
Sep 26 03:58:46 dist-test-slave-nhpw krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395126, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260926 03:58:46.927325 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:46.921372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35627 (local address 127.0.0.1:47052)
0926 03:58:46.921421 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:46.921431 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:46.921467 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:46.921618 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:46.921618 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:46.921721 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:46.921765 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:46.921766 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:46.922430 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:46.922430 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:46.922913 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:46.922914 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:46.923256 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:46.923257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:46.923303 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:46.923644 (+ 341us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:46.923648 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:46.924783 (+ 1135us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:58:46.926472 (+ 1689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:46.926472 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:46.926473 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:46.926681 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:46.926889 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:46.926889 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:46.926890 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:46.926936 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:46.927119 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:58:46.927120 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:58:46.927150 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 03:58:46.927203 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 03:58:46.927266 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 03:58:46.927342 7153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:46.921457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47052 (local address 127.0.98.61:35627)
0926 03:58:46.921494 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:46.921496 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:46.921503 (+ 7us) server_negotiation.cc:407] Connection header received
0926 03:58:46.921526 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:46.921526 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:46.921539 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:46.921611 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:46.921864 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:46.922363 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:46.922989 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:46.923191 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:46.923249 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:46.924860 (+ 1611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:46.924861 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:46.924862 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:46.924868 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:46.926400 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:46.926747 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:46.926747 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:46.926748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:46.926799 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:46.926974 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:46.926975 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:46.926975 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:46.927066 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:46.927120 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:46.927221 (+ 101us) server_negotiation.cc:299] Negotiation successful
0926 03:58:46.927283 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 03:58:46.927778 7224 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34077 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 03:58:46 dist-test-slave-nhpw krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395126, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260926 03:58:46.934268 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:46.928129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34077 (local address 127.0.0.1:45380)
0926 03:58:46.928175 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:46.928191 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:46.928225 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:46.928330 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:46.928330 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:46.928428 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:46.928452 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:46.928453 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:46.929246 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:46.929248 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:46.929758 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:46.929760 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:46.930162 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:46.930163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:46.930197 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:46.930559 (+ 362us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:46.930562 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:46.931780 (+ 1218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:58:46.933339 (+ 1559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:46.933340 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:46.933340 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:46.933495 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:46.933689 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:46.933690 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:46.933691 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:46.933741 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:46.933971 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:58:46.933971 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:58:46.934005 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 03:58:46.934072 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 03:58:46.934163 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 03:58:46.934321 7226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:46.928198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45380 (local address 127.0.98.60:34077)
0926 03:58:46.928232 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:46.928233 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:46.928242 (+ 9us) server_negotiation.cc:407] Connection header received
0926 03:58:46.928261 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:46.928262 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:46.928270 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:46.928326 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:46.928507 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:46.929165 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:46.929829 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:46.930099 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:46.930153 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:46.931874 (+ 1721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:46.931875 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:46.931875 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:46.931881 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:46.933243 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:46.933578 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:46.933578 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:46.933579 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:46.933628 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:46.933811 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:46.933812 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:46.933812 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:46.933908 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:46.933978 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:46.934107 (+ 129us) server_negotiation.cc:299] Negotiation successful
0926 03:58:46.934210 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":18}
I20260926 03:58:46.936017 7224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e84368bd28c54795852888b3d9163a75: Bootstrap starting.
I20260926 03:58:46.936933 7224 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e84368bd28c54795852888b3d9163a75: Neither blocks nor log segments found. Creating new log.
I20260926 03:58:46.937212 7224 log.cc:824] T 00000000000000000000000000000000 P e84368bd28c54795852888b3d9163a75: Log is configured to *not* fsync() on all Append() calls
I20260926 03:58:46.937906 7224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e84368bd28c54795852888b3d9163a75: No bootstrap required, opened a new log
I20260926 03:58:46.939237 7224 raft_consensus.cc:348] T 00000000000000000000000000000000 P e84368bd28c54795852888b3d9163a75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aeea5ec3a114f8c94fd2d2012fe312b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41421 } } peers { permanent_uuid: "3f6ed0be592b4c6da0be1ae5feb24ff1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35627 } } peers { permanent_uuid: "e84368bd28c54795852888b3d9163a75" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34077 } }
I20260926 03:58:46.939353 7224 raft_consensus.cc:374] T 00000000000000000000000000000000 P e84368bd28c54795852888b3d9163a75 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:58:46.939374 7224 raft_consensus.cc:729] T 00000000000000000000000000000000 P e84368bd28c54795852888b3d9163a75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e84368bd28c54795852888b3d9163a75, State: Initialized, Role: FOLLOWER
I20260926 03:58:46.939518 7224 consensus_queue.cc:260] T 00000000000000000000000000000000 P e84368bd28c54795852888b3d9163a75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aeea5ec3a114f8c94fd2d2012fe312b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41421 } } peers { permanent_uuid: "3f6ed0be592b4c6da0be1ae5feb24ff1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35627 } } peers { permanent_uuid: "e84368bd28c54795852888b3d9163a75" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34077 } }
I20260926 03:58:46.939833 7224 sys_catalog.cc:564] T 00000000000000000000000000000000 P e84368bd28c54795852888b3d9163a75 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:58:46.939791 7229 sys_catalog.cc:455] T 00000000000000000000000000000000 P e84368bd28c54795852888b3d9163a75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0aeea5ec3a114f8c94fd2d2012fe312b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41421 } } peers { permanent_uuid: "3f6ed0be592b4c6da0be1ae5feb24ff1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35627 } } peers { permanent_uuid: "e84368bd28c54795852888b3d9163a75" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34077 } } }
I20260926 03:58:46.939896 7229 sys_catalog.cc:458] T 00000000000000000000000000000000 P e84368bd28c54795852888b3d9163a75 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 03:58:46.941520 7240 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e84368bd28c54795852888b3d9163a75: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 03:58:46.941612 7240 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 03:58:46.943831 7156 master_runner.cc:428] Detected that this master e84368bd28c54795852888b3d9163a75 is joining an existing cluster
I20260926 03:58:46.943910 7156 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:34077
I20260926 03:58:46.952381 7247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:46.944887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41421 (local address 127.0.0.1:47564)
0926 03:58:46.945143 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:46.945153 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:46.945236 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:46.945420 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:46.945421 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:46.945701 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:46.945804 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:46.945806 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:46.946519 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:46.946520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:46.947487 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:46.947489 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:46.947892 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:46.947892 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:46.947953 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:46.948413 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:46.948417 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:46.948778 (+ 361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:58:46.950969 (+ 2191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:46.950970 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:46.950971 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:46.951193 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:46.951692 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:46.951693 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:46.951694 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:46.951761 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:46.952049 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:58:46.952050 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:58:46.952101 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 03:58:46.952178 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 03:58:46.952277 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":2,"thread_start_us":88,"threads_started":1}
I20260926 03:58:46.952389 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:46.944914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47564 (local address 127.0.98.62:41421)
0926 03:58:46.944975 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:46.944977 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:46.945208 (+ 231us) server_negotiation.cc:407] Connection header received
0926 03:58:46.945316 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:46.945317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:46.945329 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:46.945361 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:46.945902 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:46.946435 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:46.947591 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:46.947794 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:46.947880 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:46.948878 (+ 998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:46.948880 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:46.948880 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:46.948886 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:46.950890 (+ 2004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:46.951258 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:46.951259 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:46.951260 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:46.951610 (+ 350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:46.951838 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:46.951839 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:46.951840 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:46.951971 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:46.952047 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:46.952207 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 03:58:46.952285 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
Sep 26 03:58:46 dist-test-slave-nhpw krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395125, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260926 03:58:46.965742 7153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:46.945244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47064 (local address 127.0.98.61:35627)
0926 03:58:46.945283 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:46.945285 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:46.948788 (+ 3503us) server_negotiation.cc:407] Connection header received
0926 03:58:46.948885 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:46.948886 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:46.948896 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:46.948932 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:46.949309 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:46.949783 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:46.950791 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:46.951075 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:46.951154 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:46.952100 (+ 946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:46.952101 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:46.952101 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:46.952108 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:46.954290 (+ 2182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:46.954711 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:46.954712 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:46.954713 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:46.954772 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:46.965029 (+ 10257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:46.965030 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:46.965030 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:46.965190 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:46.965281 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:46.965530 (+ 249us) server_negotiation.cc:299] Negotiation successful
0926 03:58:46.965639 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 03:58:46.966018 7246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:46.945109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35627 (local address 127.0.0.1:47064)
0926 03:58:46.948723 (+ 3614us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:46.948734 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:46.948813 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:46.948992 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:46.948992 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:46.949182 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:46.949244 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:46.949246 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:46.949863 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:46.949864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:46.950698 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:46.950700 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:46.951152 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:46.951153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:46.951215 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:46.951624 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:46.951629 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:46.952002 (+ 373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:58:46.954419 (+ 2417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:46.954420 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:46.954421 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:46.954638 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:46.954849 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:46.954850 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:46.954850 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:46.964932 (+ 10082us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:46.965309 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:58:46.965310 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:58:46.965356 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 03:58:46.965427 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 03:58:46.965919 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3597,"thread_start_us":63,"threads_started":1}
I20260926 03:58:46.968258 7056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:46.954611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34077 (local address 127.0.0.1:45394)
0926 03:58:46.954692 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:46.954702 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:46.954734 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:46.955086 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:46.955087 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:46.955235 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:46.955266 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:46.955267 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:46.955895 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:46.955896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:46.956397 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:46.956399 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:46.956861 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:46.956862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:46.956926 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:46.957333 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:46.957337 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:46.958453 (+ 1116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:58:46.967134 (+ 8681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:46.967135 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:46.967135 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:46.967366 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:46.967595 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:46.967596 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:46.967596 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:46.967658 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:46.967976 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:58:46.967976 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:58:46.968019 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 03:58:46.968066 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 03:58:46.968169 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64}
I20260926 03:58:46.968667 7249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:46.954800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45394 (local address 127.0.98.60:34077)
0926 03:58:46.954953 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:46.954954 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:46.954960 (+ 6us) server_negotiation.cc:407] Connection header received
0926 03:58:46.954991 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:46.954991 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:46.955006 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:46.955082 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:46.955341 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:46.955814 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:46.956491 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:46.956778 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:46.956860 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:46.958537 (+ 1677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:46.958538 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:46.958538 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:46.958545 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:46.967036 (+ 8491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:46.967462 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:46.967463 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:46.967464 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:46.967528 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:46.967743 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:46.967744 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:46.967744 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:46.967850 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:46.968397 (+ 547us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:46.968438 (+ 41us) server_negotiation.cc:299] Negotiation successful
0926 03:58:46.968510 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":113,"threads_started":1}
I20260926 03:58:46.969605 7248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:46.945012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34077 (local address 127.0.0.1:45392)
0926 03:58:46.945203 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:46.945212 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:46.945272 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:46.945432 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:46.945433 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:46.945704 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:46.945804 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:46.945806 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:46.946673 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:46.946673 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:46.947487 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:46.947489 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:46.947970 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:46.947971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:46.948029 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:46.966017 (+ 17988us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:46.966022 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:46.966455 (+ 433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:58:46.968256 (+ 1801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:46.968256 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:46.968257 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:46.968459 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:46.968662 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:46.968663 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:46.968663 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:46.968757 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:46.969330 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:58:46.969330 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:58:46.969361 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 03:58:46.969417 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 03:58:46.969505 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":9,"thread_start_us":95,"threads_started":1}
I20260926 03:58:46.970041 7226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:46.944931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45392 (local address 127.0.98.60:34077)
0926 03:58:46.944967 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:46.944969 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:46.945253 (+ 284us) server_negotiation.cc:407] Connection header received
0926 03:58:46.945350 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:46.945351 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:46.945361 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:46.945428 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:46.945879 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:46.946593 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:46.947567 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:46.947855 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:46.947924 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:46.966543 (+ 18619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:46.966544 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:46.966544 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:46.966552 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:46.968168 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:46.968525 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:46.968526 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:46.968526 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:46.968569 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:46.969121 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:46.969122 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:46.969122 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:46.969244 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:46.969309 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:46.969895 (+ 586us) server_negotiation.cc:299] Negotiation successful
0926 03:58:46.969964 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 03:58:46.970060 6498 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:34077: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "e84368bd28c54795852888b3d9163a75"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 34077
}
attrs {
promote: true
}
}
dest_uuid: "0aeea5ec3a114f8c94fd2d2012fe312b"
cas_config_opid_index: 5
I20260926 03:58:46.970585 6498 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0aeea5ec3a114f8c94fd2d2012fe312b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0aeea5ec3a114f8c94fd2d2012fe312b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41421 } } peers { permanent_uuid: "3f6ed0be592b4c6da0be1ae5feb24ff1" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35627 } attrs { promote: true } } peers { permanent_uuid: "e84368bd28c54795852888b3d9163a75" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34077 } attrs { promote: true } }
I20260926 03:58:46.971333 7057 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0aeea5ec3a114f8c94fd2d2012fe312b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER e84368bd28c54795852888b3d9163a75 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0aeea5ec3a114f8c94fd2d2012fe312b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41421 } } peers { permanent_uuid: "3f6ed0be592b4c6da0be1ae5feb24ff1" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35627 } attrs { promote: true } } peers { permanent_uuid: "e84368bd28c54795852888b3d9163a75" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34077 } attrs { promote: true } } }
I20260926 03:58:46.971781 7198 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e84368bd28c54795852888b3d9163a75 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:58:46.972468 7057 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:34077
I20260926 03:58:46.972544 7057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0aeea5ec3a114f8c94fd2d2012fe312b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0aeea5ec3a114f8c94fd2d2012fe312b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0aeea5ec3a114f8c94fd2d2012fe312b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41421 } } peers { permanent_uuid: "3f6ed0be592b4c6da0be1ae5feb24ff1" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35627 } attrs { promote: true } } peers { permanent_uuid: "e84368bd28c54795852888b3d9163a75" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34077 } attrs { promote: true } } }
I20260926 03:58:46.972774 7198 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e84368bd28c54795852888b3d9163a75 [term 1 FOLLOWER]: Refusing update from remote peer 0aeea5ec3a114f8c94fd2d2012fe312b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 03:58:46.972890 7156 master.cc:537] Master@127.0.98.60:34077 shutting down...
I20260926 03:58:46.973546 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 6560
I20260926 03:58:46.975550 7057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0aeea5ec3a114f8c94fd2d2012fe312b [sys.catalog]: This master's current role is: LEADER
I20260926 03:58:46.976783 7056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:46.973141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35627 (local address 127.0.0.1:47068)
0926 03:58:46.973184 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:46.973194 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:46.973233 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:46.973429 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:46.973430 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:46.973432 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:58:46.973471 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:46.973473 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:46.974434 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:46.974435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:46.975618 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:46.975621 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:46.976473 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:46.976474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:46.976545 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:46.976569 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 03:58:46.976674 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 03:58:46.976714 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260926 03:58:46.976790 7153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:46.973204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47068 (local address 127.0.98.61:35627)
0926 03:58:46.973241 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:46.973243 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:46.973252 (+ 9us) server_negotiation.cc:407] Connection header received
0926 03:58:46.973293 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:46.973293 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:46.973314 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:46.973371 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:58:46.973544 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:46.974279 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:46.975697 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:46.976384 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:46.976464 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:46.976544 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:46.976659 (+ 115us) server_negotiation.cc:299] Negotiation successful
0926 03:58:46.976699 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 03:58:46.980351 7112 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3f6ed0be592b4c6da0be1ae5feb24ff1 [term 1 LEARNER]: Refusing update from remote peer 0aeea5ec3a114f8c94fd2d2012fe312b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 03:58:46.973295 7250 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0aeea5ec3a114f8c94fd2d2012fe312b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e84368bd28c54795852888b3d9163a75" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34077 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:58:46.980676 7250 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0aeea5ec3a114f8c94fd2d2012fe312b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f6ed0be592b4c6da0be1ae5feb24ff1" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35627 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260926 03:58:46.980836 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:37187
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:41421
--webserver_port=43513
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:41421,127.0.98.61:35627
--tserver_master_addrs=127.0.98.62:41421,127.0.98.61:35627,127.0.98.60:34077 with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:58:46.981191 7156 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e84368bd28c54795852888b3d9163a75 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:58:46.981287 7156 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e84368bd28c54795852888b3d9163a75 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:58:46.981338 7156 tablet_replica.cc:330] T 00000000000000000000000000000000 P e84368bd28c54795852888b3d9163a75: stopping tablet replica
W20260926 03:58:46.981712 6485 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260926 03:58:46.982200 7112 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3f6ed0be592b4c6da0be1ae5feb24ff1 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER e84368bd28c54795852888b3d9163a75 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0aeea5ec3a114f8c94fd2d2012fe312b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41421 } } peers { permanent_uuid: "3f6ed0be592b4c6da0be1ae5feb24ff1" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35627 } attrs { promote: true } } peers { permanent_uuid: "e84368bd28c54795852888b3d9163a75" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34077 } attrs { promote: true } } }
I20260926 03:58:46.983390 7057 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 0aeea5ec3a114f8c94fd2d2012fe312b: attempting to promote NON_VOTER 3f6ed0be592b4c6da0be1ae5feb24ff1 to VOTER
I20260926 03:58:46.983546 7141 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f6ed0be592b4c6da0be1ae5feb24ff1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0aeea5ec3a114f8c94fd2d2012fe312b. Latest consensus state: current_term: 1 leader_uuid: "0aeea5ec3a114f8c94fd2d2012fe312b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0aeea5ec3a114f8c94fd2d2012fe312b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41421 } } peers { permanent_uuid: "3f6ed0be592b4c6da0be1ae5feb24ff1" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35627 } attrs { promote: true } } peers { permanent_uuid: "e84368bd28c54795852888b3d9163a75" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34077 } attrs { promote: true } } }
I20260926 03:58:46.983628 7141 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f6ed0be592b4c6da0be1ae5feb24ff1 [sys.catalog]: This master's current role is: LEARNER
I20260926 03:58:46.983582 7057 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0aeea5ec3a114f8c94fd2d2012fe312b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0aeea5ec3a114f8c94fd2d2012fe312b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41421 } } peers { permanent_uuid: "3f6ed0be592b4c6da0be1ae5feb24ff1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35627 } attrs { promote: false } } peers { permanent_uuid: "e84368bd28c54795852888b3d9163a75" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34077 } attrs { promote: true } }
I20260926 03:58:46.983664 7141 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f6ed0be592b4c6da0be1ae5feb24ff1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0aeea5ec3a114f8c94fd2d2012fe312b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "0aeea5ec3a114f8c94fd2d2012fe312b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41421 } } peers { permanent_uuid: "3f6ed0be592b4c6da0be1ae5feb24ff1" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 35627 } attrs { promote: true } } peers { permanent_uuid: "e84368bd28c54795852888b3d9163a75" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34077 } attrs { promote: true } } }
I20260926 03:58:46.983703 7141 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f6ed0be592b4c6da0be1ae5feb24ff1 [sys.catalog]: This master's current role is: LEARNER
I20260926 03:58:46.984170 7056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:46.984036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34077 (local address 127.0.0.1:45406)
0926 03:58:46.984091 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:46.984136 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 03:58:46.984469 7112 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3f6ed0be592b4c6da0be1ae5feb24ff1 [term 1 LEARNER]: Refusing update from remote peer 0aeea5ec3a114f8c94fd2d2012fe312b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260926 03:58:46.984575 7056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:46.984384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34077 (local address 127.0.0.1:45416)
0926 03:58:46.984527 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:46.984551 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
W20260926 03:58:46.984678 6485 consensus_peers.cc:487] T 00000000000000000000000000000000 P 0aeea5ec3a114f8c94fd2d2012fe312b -> Peer e84368bd28c54795852888b3d9163a75 (127.0.98.60:34077): Couldn't send request to peer e84368bd28c54795852888b3d9163a75. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34077: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 03:58:46.984690 7250 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0aeea5ec3a114f8c94fd2d2012fe312b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f6ed0be592b4c6da0be1ae5feb24ff1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35627 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260926 03:58:46.991940 7058 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0aeea5ec3a114f8c94fd2d2012fe312b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 3f6ed0be592b4c6da0be1ae5feb24ff1 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0aeea5ec3a114f8c94fd2d2012fe312b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41421 } } peers { permanent_uuid: "3f6ed0be592b4c6da0be1ae5feb24ff1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35627 } attrs { promote: false } } peers { permanent_uuid: "e84368bd28c54795852888b3d9163a75" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34077 } attrs { promote: true } } }
I20260926 03:58:46.992839 7112 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3f6ed0be592b4c6da0be1ae5feb24ff1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 3f6ed0be592b4c6da0be1ae5feb24ff1 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0aeea5ec3a114f8c94fd2d2012fe312b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41421 } } peers { permanent_uuid: "3f6ed0be592b4c6da0be1ae5feb24ff1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35627 } attrs { promote: false } } peers { permanent_uuid: "e84368bd28c54795852888b3d9163a75" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34077 } attrs { promote: true } } }
I20260926 03:58:46.992908 7058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0aeea5ec3a114f8c94fd2d2012fe312b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0aeea5ec3a114f8c94fd2d2012fe312b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0aeea5ec3a114f8c94fd2d2012fe312b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41421 } } peers { permanent_uuid: "3f6ed0be592b4c6da0be1ae5feb24ff1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35627 } attrs { promote: false } } peers { permanent_uuid: "e84368bd28c54795852888b3d9163a75" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34077 } attrs { promote: true } } }
I20260926 03:58:46.992980 7058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0aeea5ec3a114f8c94fd2d2012fe312b [sys.catalog]: This master's current role is: LEADER
I20260926 03:58:46.993027 7058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0aeea5ec3a114f8c94fd2d2012fe312b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0aeea5ec3a114f8c94fd2d2012fe312b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0aeea5ec3a114f8c94fd2d2012fe312b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41421 } } peers { permanent_uuid: "3f6ed0be592b4c6da0be1ae5feb24ff1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35627 } attrs { promote: false } } peers { permanent_uuid: "e84368bd28c54795852888b3d9163a75" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34077 } attrs { promote: true } } }
I20260926 03:58:46.993081 7058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0aeea5ec3a114f8c94fd2d2012fe312b [sys.catalog]: This master's current role is: LEADER
I20260926 03:58:46.993777 7254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f6ed0be592b4c6da0be1ae5feb24ff1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0aeea5ec3a114f8c94fd2d2012fe312b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0aeea5ec3a114f8c94fd2d2012fe312b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41421 } } peers { permanent_uuid: "3f6ed0be592b4c6da0be1ae5feb24ff1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35627 } attrs { promote: false } } peers { permanent_uuid: "e84368bd28c54795852888b3d9163a75" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34077 } attrs { promote: true } } }
I20260926 03:58:46.993862 7254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f6ed0be592b4c6da0be1ae5feb24ff1 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:58:46.995398 7156 master.cc:552] Master@127.0.98.60:34077 shutdown complete.
I20260926 03:58:46.995448 7156 master_runner.cc:305] Clearing existing system tablet
I20260926 03:58:46.995575 7156 fs_manager.cc:359] Metadata directory not provided
I20260926 03:58:46.995621 7156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:58:46.996346 7156 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 03:58:46.996883 7256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:46.997027 7156 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:58:46.997094 7156 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:46.997118 7156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/master-2/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/master-2/wal
uuid: "e84368bd28c54795852888b3d9163a75"
format_stamp: "Formatted at 2026-09-26 03:58:46 on dist-test-slave-nhpw"
server_key: "c25e61eb5e7a6127e8e669dca1a3d9a1"
server_key_iv: "042d7203e66df8902fdfe5871e78449c"
server_key_version: "encryptionkey@0"
I20260926 03:58:46.997881 7156 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P e84368bd28c54795852888b3d9163a75: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:58:46.999260 7156 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P e84368bd28c54795852888b3d9163a75: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 03:58:46.999330 7156 log.cc:1196] T 00000000000000000000000000000000 P e84368bd28c54795852888b3d9163a75: Deleting WAL directory at /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260926 03:58:46.999521 7156 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P e84368bd28c54795852888b3d9163a75: Deleting consensus metadata
I20260926 03:58:46.999642 7156 master_runner.cc:315] Copying system tablet from 127.0.98.62:41421
I20260926 03:58:47.000236 7156 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P e84368bd28c54795852888b3d9163a75: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:41421
I20260926 03:58:47.007416 7262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:47.000492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41421 (local address 127.0.0.1:47572)
0926 03:58:47.000604 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:47.000614 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:47.000688 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:47.000838 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:47.000839 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:47.001041 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:47.001114 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:47.001116 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:47.002058 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:47.002059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:47.002901 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:47.002904 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:47.003404 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:47.003405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:47.003468 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:47.003993 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:47.003998 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:47.004374 (+ 376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:58:47.006338 (+ 1964us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:47.006339 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:47.006340 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:47.006547 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:47.006760 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:47.006760 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:47.006761 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:47.006821 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:47.007099 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:58:47.007100 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:58:47.007146 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 03:58:47.007212 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 03:58:47.007310 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":63,"threads_started":1}
I20260926 03:58:47.007416 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:47.000553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47572 (local address 127.0.98.62:41421)
0926 03:58:47.000597 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:47.000599 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:47.000674 (+ 75us) server_negotiation.cc:407] Connection header received
0926 03:58:47.000758 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:47.000759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:47.000769 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:47.000826 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:47.001196 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:47.001969 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:47.003001 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:47.003319 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:47.003395 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:47.004468 (+ 1073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:47.004469 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:47.004469 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:47.004477 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:47.006253 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:47.006627 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:47.006628 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:47.006628 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:47.006689 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:47.006887 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:47.006887 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:47.006888 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:47.007016 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:47.007087 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:47.007252 (+ 165us) server_negotiation.cc:299] Negotiation successful
0926 03:58:47.007327 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 03:58:47.008049 6518 tablet_copy_service.cc:140] P 0aeea5ec3a114f8c94fd2d2012fe312b: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer e84368bd28c54795852888b3d9163a75 ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:47572)
I20260926 03:58:47.008116 6518 tablet_copy_service.cc:161] P 0aeea5ec3a114f8c94fd2d2012fe312b: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer e84368bd28c54795852888b3d9163a75 at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:47572: session id = e84368bd28c54795852888b3d9163a75-00000000000000000000000000000000
I20260926 03:58:47.008661 6518 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 0aeea5ec3a114f8c94fd2d2012fe312b: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 03:58:47.009114 7156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 03:58:47.010324 7156 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P e84368bd28c54795852888b3d9163a75: tablet copy: Starting download of 0 data blocks...
I20260926 03:58:47.010463 7156 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P e84368bd28c54795852888b3d9163a75: tablet copy: Starting download of 1 WAL segments...
I20260926 03:58:47.011116 7156 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P e84368bd28c54795852888b3d9163a75: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 03:58:47.011956 6518 tablet_copy_service.cc:342] P 0aeea5ec3a114f8c94fd2d2012fe312b: Request end of tablet copy session e84368bd28c54795852888b3d9163a75-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:47572
I20260926 03:58:47.012002 6518 tablet_copy_service.cc:433] P 0aeea5ec3a114f8c94fd2d2012fe312b: ending tablet copy session e84368bd28c54795852888b3d9163a75-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer e84368bd28c54795852888b3d9163a75
I20260926 03:58:47.014614 7156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:58:47.017218 7263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:47.020831 7264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:47.021600 7266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:47.021821 7156 server_base.cc:1034] running on GCE node
I20260926 03:58:47.021922 7156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:58:47.022049 7156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:58:47.023197 7156 hybrid_clock.cc:648] HybridClock initialized: now 1790395127023174 us; error 39 us; skew 500 ppm
Sep 26 03:58:47 dist-test-slave-nhpw krb5kdc[6453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395127, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:58:47.024749 7156 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260926 03:58:47.025005 7156 webserver.cc:460] Webserver started at http://127.0.98.60:42839/ using document root <none> and password file <none>
I20260926 03:58:47.025134 7156 fs_manager.cc:359] Metadata directory not provided
I20260926 03:58:47.025171 7156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:58:47.025903 7156 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 03:58:47.026502 7273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:47.026710 7156 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 03:58:47.026764 7156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/master-2/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/master-2/wal
uuid: "e84368bd28c54795852888b3d9163a75"
format_stamp: "Formatted at 2026-09-26 03:58:46 on dist-test-slave-nhpw"
server_key: "c25e61eb5e7a6127e8e669dca1a3d9a1"
server_key_iv: "042d7203e66df8902fdfe5871e78449c"
server_key_version: "encryptionkey@0"
I20260926 03:58:47.026822 7156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:58:47.034590 7156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:58:47.036489 7156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:58:47.039521 7156 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:34077
I20260926 03:58:47.039903 7156 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/master-2/data/info.pb
I20260926 03:58:47.040884 7326 sys_catalog.cc:263] Verifying existing consensus state
I20260926 03:58:47.041399 7326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e84368bd28c54795852888b3d9163a75: Bootstrap starting.
I20260926 03:58:47.042798 7325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:34077 every 8 connection(s)
I20260926 03:58:47.044795 7326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e84368bd28c54795852888b3d9163a75: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 03:58:47.045039 7326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e84368bd28c54795852888b3d9163a75: Bootstrap complete.
I20260926 03:58:47.045262 7326 raft_consensus.cc:348] T 00000000000000000000000000000000 P e84368bd28c54795852888b3d9163a75 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0aeea5ec3a114f8c94fd2d2012fe312b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41421 } } peers { permanent_uuid: "3f6ed0be592b4c6da0be1ae5feb24ff1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35627 } attrs { promote: false } } peers { permanent_uuid: "e84368bd28c54795852888b3d9163a75" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34077 } attrs { promote: true } }
I20260926 03:58:47.045363 7326 raft_consensus.cc:729] T 00000000000000000000000000000000 P e84368bd28c54795852888b3d9163a75 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e84368bd28c54795852888b3d9163a75, State: Initialized, Role: LEARNER
I20260926 03:58:47.045403 7326 consensus_queue.cc:260] T 00000000000000000000000000000000 P e84368bd28c54795852888b3d9163a75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0aeea5ec3a114f8c94fd2d2012fe312b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41421 } } peers { permanent_uuid: "3f6ed0be592b4c6da0be1ae5feb24ff1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35627 } attrs { promote: false } } peers { permanent_uuid: "e84368bd28c54795852888b3d9163a75" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34077 } attrs { promote: true } }
I20260926 03:58:47.045599 7326 sys_catalog.cc:564] T 00000000000000000000000000000000 P e84368bd28c54795852888b3d9163a75 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:58:47.045609 7329 sys_catalog.cc:455] T 00000000000000000000000000000000 P e84368bd28c54795852888b3d9163a75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0aeea5ec3a114f8c94fd2d2012fe312b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41421 } } peers { permanent_uuid: "3f6ed0be592b4c6da0be1ae5feb24ff1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35627 } attrs { promote: false } } peers { permanent_uuid: "e84368bd28c54795852888b3d9163a75" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34077 } attrs { promote: true } } }
I20260926 03:58:47.045820 7329 sys_catalog.cc:458] T 00000000000000000000000000000000 P e84368bd28c54795852888b3d9163a75 [sys.catalog]: This master's current role is: LEARNER
I20260926 03:58:47.047194 7340 catalog_manager.cc:1259] Loaded cluster ID: a989010329ac4d63b6f8b327d43ccef5
I20260926 03:58:47.047317 7340 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e84368bd28c54795852888b3d9163a75: loading cluster ID for follower catalog manager: success
I20260926 03:58:47.048585 7340 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e84368bd28c54795852888b3d9163a75: acquiring CA information for follower catalog manager: success
I20260926 03:58:47.048981 7340 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e84368bd28c54795852888b3d9163a75: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 03:58:47.077523 7252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:58:47.077740 7252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:58:47.077782 7252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:58:47.078935 7252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:58:47.079002 7252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:58:47.079047 7252 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 03:58:47.080497 7252 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:37187
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=43513
--tserver_master_addrs=127.0.98.62:41421,127.0.98.61:35627,127.0.98.60:34077
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7252
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:58:47.080776 7252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:58:47.081000 7252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:58:47.086723 7346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:47.086778 7345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:47.086928 7252 server_base.cc:1034] running on GCE node
W20260926 03:58:47.086723 7348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:47.087203 7252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:58:47.087481 7252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:58:47.088685 7252 hybrid_clock.cc:648] HybridClock initialized: now 1790395127088670 us; error 30 us; skew 500 ppm
Sep 26 03:58:47 dist-test-slave-nhpw krb5kdc[6453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395127, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:58:47.091346 7252 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260926 03:58:47.091853 7252 webserver.cc:460] Webserver started at http://127.0.98.1:43513/ using document root <none> and password file <none>
I20260926 03:58:47.092082 7252 fs_manager.cc:359] Metadata directory not provided
I20260926 03:58:47.092185 7252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:58:47.093662 7252 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 03:58:47.094276 7355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:47.094487 7252 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 03:58:47.094576 7252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-0/wal
uuid: "226f08ac9eb34274ae439b88cea9050c"
format_stamp: "Formatted at 2026-09-26 03:58:46 on dist-test-slave-nhpw"
server_key: "78f7b1ebb2c68787fbbbf09bbf0feeb9"
server_key_iv: "67d2f7c752cb906ace8a4ac5738b2a82"
server_key_version: "encryptionkey@0"
I20260926 03:58:47.094982 7252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:58:47.115329 7252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:58:47.117520 7252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:58:47.117635 7252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:58:47.117825 7252 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:58:47.118147 7252 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:58:47.118183 7252 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:47.118206 7252 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:58:47.118255 7252 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:47.124197 7252 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37187
I20260926 03:58:47.124270 7468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37187 every 8 connection(s)
I20260926 03:58:47.124648 7252 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
I20260926 03:58:47.127473 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 7252
I20260926 03:58:47.127653 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 6694
Sep 26 03:58:47 dist-test-slave-nhpw krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395127, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 03:58:47.132952 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:43115
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:41421
--webserver_port=38885
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:41421,127.0.98.61:35627
--tserver_master_addrs=127.0.98.62:41421,127.0.98.61:35627,127.0.98.60:34077 with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 03:58:47 dist-test-slave-nhpw krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395127, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 26 03:58:47 dist-test-slave-nhpw krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395127, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260926 03:58:47.138295 7153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:47.125417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40909 (local address 127.0.98.61:35627)
0926 03:58:47.125463 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:47.125465 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:47.125656 (+ 191us) server_negotiation.cc:407] Connection header received
0926 03:58:47.126744 (+ 1088us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:47.126745 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:47.126758 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:47.126812 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:47.129326 (+ 2514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:47.130069 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:47.130950 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:47.131242 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:47.131317 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:47.134567 (+ 3250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:47.134569 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:47.134569 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:47.134578 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:47.136517 (+ 1939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:47.137133 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:47.137134 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:47.137134 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:47.137194 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:47.137525 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:47.137526 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:47.137527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:47.137659 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:47.137715 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:47.138097 (+ 382us) server_negotiation.cc:299] Negotiation successful
0926 03:58:47.138183 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 03:58:47.138603 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:47.125772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58009 (local address 127.0.98.62:41421)
0926 03:58:47.125841 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:47.125843 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:47.126858 (+ 1015us) server_negotiation.cc:407] Connection header received
0926 03:58:47.126933 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:47.126934 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:47.126944 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:47.126993 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:47.128403 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:47.128993 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:47.130670 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:47.130963 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:47.131036 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:47.134609 (+ 3573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:47.134610 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:47.134610 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:47.134618 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:47.136697 (+ 2079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:47.137133 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:47.137134 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:47.137134 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:47.137201 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:47.137510 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:47.137511 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:47.137512 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:47.137913 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:47.137961 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:47.138189 (+ 228us) server_negotiation.cc:299] Negotiation successful
0926 03:58:47.138508 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 03:58:47.138839 7474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:47.125381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35627 (local address 127.0.98.1:40909)
0926 03:58:47.125544 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:47.125578 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:47.126627 (+ 1049us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:47.126895 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:47.126899 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:47.127421 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:47.129239 (+ 1818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:47.129242 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:47.130165 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:47.130166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:47.130871 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:47.130873 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:47.131329 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:47.131330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:47.131398 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:47.131894 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:47.131898 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:47.134442 (+ 2544us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:58:47.136617 (+ 2175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:47.136624 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:47.136627 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:47.137038 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:47.137275 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:47.137275 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:47.137276 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:47.137430 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:47.137749 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:58:47.137755 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:58:47.137924 (+ 169us) client_negotiation.cc:769] Sending connection context
0926 03:58:47.138398 (+ 474us) client_negotiation.cc:240] Negotiation successful
0926 03:58:47.138594 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"mutex_wait_us":2,"thread_start_us":97,"threads_started":1}
I20260926 03:58:47.138936 7473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:47.125679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41421 (local address 127.0.98.1:58009)
0926 03:58:47.126811 (+ 1132us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:47.126820 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:47.126866 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:47.126995 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:47.126996 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:47.127460 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:47.128302 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:47.128307 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:47.129102 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:47.129103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:47.130583 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:47.130585 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:47.131046 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:47.131047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:47.131176 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:47.131745 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:47.131755 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:47.134098 (+ 2343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:58:47.136775 (+ 2677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:47.136776 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:47.136777 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:47.137053 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:47.137275 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:47.137275 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:47.137276 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:47.137430 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:47.138022 (+ 592us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:58:47.138023 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:58:47.138073 (+ 50us) client_negotiation.cc:769] Sending connection context
0926 03:58:47.138718 (+ 645us) client_negotiation.cc:240] Negotiation successful
0926 03:58:47.138803 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1114,"mutex_wait_us":146,"thread_start_us":50,"threads_started":1}
I20260926 03:58:47.139626 7471 heartbeater.cc:347] Connected to a master server at 127.0.98.61:35627
I20260926 03:58:47.139635 7470 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41421
I20260926 03:58:47.139745 7471 heartbeater.cc:464] Registering TS with master...
I20260926 03:58:47.139745 7470 heartbeater.cc:464] Registering TS with master...
I20260926 03:58:47.139963 7471 heartbeater.cc:511] Master 127.0.98.61:35627 requested a full tablet report, sending...
I20260926 03:58:47.139963 7470 heartbeater.cc:511] Master 127.0.98.62:41421 requested a full tablet report, sending...
I20260926 03:58:47.140470 6498 ts_manager.cc:194] Re-registered known tserver with Master: 226f08ac9eb34274ae439b88cea9050c (127.0.98.1:37187)
I20260926 03:58:47.140573 7102 ts_manager.cc:194] Registered new tserver with Master: 226f08ac9eb34274ae439b88cea9050c (127.0.98.1:37187)
I20260926 03:58:47.140967 6498 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58009
I20260926 03:58:47.142244 7475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:47.125381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34077 (local address 127.0.98.1:49445)
0926 03:58:47.125893 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:47.125903 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:47.126640 (+ 737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:47.126895 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:47.126899 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:47.127454 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:47.132731 (+ 5277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:47.132734 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:47.133759 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:47.133760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:47.134570 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:47.134572 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:47.135046 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:47.135048 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:47.135125 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:47.135635 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:47.135641 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:47.137281 (+ 1640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:58:47.140688 (+ 3407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:47.140689 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:47.140689 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:47.140966 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:47.141326 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:47.141327 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:47.141327 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:47.141388 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:47.141701 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:58:47.141702 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:58:47.141798 (+ 96us) client_negotiation.cc:769] Sending connection context
0926 03:58:47.141872 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 03:58:47.141971 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"mutex_wait_us":54,"thread_start_us":98,"threads_started":1}
I20260926 03:58:47.142568 7476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:47.125787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49445 (local address 127.0.98.60:34077)
0926 03:58:47.125894 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:47.125896 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:47.125951 (+ 55us) server_negotiation.cc:407] Connection header received
0926 03:58:47.126729 (+ 778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:47.126730 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:47.126743 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:47.126822 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:47.132830 (+ 6008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:47.133674 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:47.134652 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:47.134962 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:47.135046 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:47.137386 (+ 2340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:47.137388 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:47.137389 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:47.137397 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:47.140484 (+ 3087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:47.141175 (+ 691us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:47.141177 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:47.141177 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:47.141243 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:47.141462 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:47.141463 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:47.141463 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:47.141604 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:47.141682 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:47.141957 (+ 275us) server_negotiation.cc:299] Negotiation successful
0926 03:58:47.142055 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":56,"threads_started":1}
I20260926 03:58:47.142946 7469 heartbeater.cc:347] Connected to a master server at 127.0.98.60:34077
I20260926 03:58:47.143006 7469 heartbeater.cc:464] Registering TS with master...
I20260926 03:58:47.143076 7469 heartbeater.cc:511] Master 127.0.98.60:34077 requested a full tablet report, sending...
I20260926 03:58:47.143419 7290 ts_manager.cc:194] Registered new tserver with Master: 226f08ac9eb34274ae439b88cea9050c (127.0.98.1:37187)
W20260926 03:58:47.218508 7477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:58:47.218765 7477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:58:47.218807 7477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:58:47.219959 7477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:58:47.220027 7477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:58:47.220072 7477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 03:58:47.221578 7477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:43115
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=38885
--tserver_master_addrs=127.0.98.62:41421,127.0.98.61:35627,127.0.98.60:34077
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:58:47.221855 7477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:58:47.222086 7477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:58:47.227205 7485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:47.227211 7483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:47.227223 7482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:47.227382 7477 server_base.cc:1034] running on GCE node
I20260926 03:58:47.227727 7477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:58:47.228005 7477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:58:47.229187 7477 hybrid_clock.cc:648] HybridClock initialized: now 1790395127229170 us; error 29 us; skew 500 ppm
Sep 26 03:58:47 dist-test-slave-nhpw krb5kdc[6453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395127, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:58:47.231786 7477 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260926 03:58:47.232257 7477 webserver.cc:460] Webserver started at http://127.0.98.2:38885/ using document root <none> and password file <none>
I20260926 03:58:47.232465 7477 fs_manager.cc:359] Metadata directory not provided
I20260926 03:58:47.232515 7477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:58:47.233883 7477 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 03:58:47.234439 7492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:47.234607 7477 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 03:58:47.234699 7477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-1/wal
uuid: "1e41eabd06d44a9ebab5ba2a76d5ea66"
format_stamp: "Formatted at 2026-09-26 03:58:46 on dist-test-slave-nhpw"
server_key: "455538b44176646187b9e6274d74fcc8"
server_key_iv: "e7d526d8b25608b1e9da2a5627b9776c"
server_key_version: "encryptionkey@0"
I20260926 03:58:47.235014 7477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:58:47.266537 7477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:58:47.269085 7477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:58:47.269207 7477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:58:47.269402 7477 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:58:47.269723 7477 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:58:47.269758 7477 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:47.269804 7477 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:58:47.269830 7477 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:47.275336 7477 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43115
I20260926 03:58:47.275415 7605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43115 every 8 connection(s)
I20260926 03:58:47.275765 7477 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
I20260926 03:58:47.279253 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 7477
I20260926 03:58:47.279404 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 6828
Sep 26 03:58:47 dist-test-slave-nhpw krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395127, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 26 03:58:47 dist-test-slave-nhpw krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395127, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 26 03:58:47 dist-test-slave-nhpw krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395127, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 03:58:47.287644 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:43813
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:41421
--webserver_port=38727
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:41421,127.0.98.61:35627
--tserver_master_addrs=127.0.98.62:41421,127.0.98.61:35627,127.0.98.60:34077 with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:58:47.288900 7153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:47.276973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50891 (local address 127.0.98.61:35627)
0926 03:58:47.277017 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:47.277019 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:47.277113 (+ 94us) server_negotiation.cc:407] Connection header received
0926 03:58:47.277505 (+ 392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:47.277506 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:47.277518 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:47.277566 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:47.278545 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:47.279067 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:47.281036 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:47.281369 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:47.281409 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:47.285772 (+ 4363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:47.285773 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:47.285773 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:47.285781 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:47.287374 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:47.287824 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:47.287825 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:47.287825 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:47.287879 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:47.288099 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:47.288099 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:47.288100 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:47.288244 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:47.288285 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:47.288679 (+ 394us) server_negotiation.cc:299] Negotiation successful
0926 03:58:47.288767 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 03:58:47.288916 7476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:47.276556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49833 (local address 127.0.98.60:34077)
0926 03:58:47.276600 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:47.276602 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:47.276892 (+ 290us) server_negotiation.cc:407] Connection header received
0926 03:58:47.277518 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:47.277520 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:47.277537 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:47.277568 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:47.278568 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:47.279343 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:47.281040 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:47.281349 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:47.281384 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:47.285002 (+ 3618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:47.285004 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:47.285009 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:47.285017 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:47.287102 (+ 2085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:47.287721 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:47.287722 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:47.287723 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:47.287781 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:47.288085 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:47.288087 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:47.288087 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:47.288243 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:47.288302 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:47.288710 (+ 408us) server_negotiation.cc:299] Negotiation successful
0926 03:58:47.288801 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 03:58:47.289532 7612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:47.276901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35627 (local address 127.0.98.2:50891)
0926 03:58:47.277024 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:47.277034 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:47.277402 (+ 368us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:47.277909 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:47.277910 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:47.278262 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:47.278451 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:47.278456 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:47.279199 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:47.279200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:47.280934 (+ 1734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:47.280937 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:47.281494 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:47.281494 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:47.281604 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:47.282298 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:47.282308 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:47.285682 (+ 3374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:58:47.287479 (+ 1797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:47.287480 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:47.287480 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:47.287736 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:47.287945 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:47.287945 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:47.287946 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:47.288030 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:47.288328 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:58:47.288333 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:58:47.288504 (+ 171us) client_negotiation.cc:769] Sending connection context
0926 03:58:47.289042 (+ 538us) client_negotiation.cc:240] Negotiation successful
0926 03:58:47.289357 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":111,"thread_start_us":66,"threads_started":1}
I20260926 03:58:47.289546 7610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:47.276551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34077 (local address 127.0.98.2:49833)
0926 03:58:47.276750 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:47.276799 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:47.277382 (+ 583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:47.277649 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:47.277655 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:47.278189 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:47.278451 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:47.278457 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:47.279427 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:47.279428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:47.280933 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:47.280936 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:47.281464 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:47.281465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:47.281602 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:47.282477 (+ 875us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:47.282482 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:47.284866 (+ 2384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:58:47.287298 (+ 2432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:47.287305 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:47.287308 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:47.287623 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:47.287853 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:47.287854 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:47.287854 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:47.288009 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:47.288334 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:58:47.288335 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:58:47.288504 (+ 169us) client_negotiation.cc:769] Sending connection context
0926 03:58:47.289050 (+ 546us) client_negotiation.cc:240] Negotiation successful
0926 03:58:47.289265 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":52,"thread_start_us":123,"threads_started":1}
I20260926 03:58:47.289639 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:47.276614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55779 (local address 127.0.98.62:41421)
0926 03:58:47.276670 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:47.276672 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:47.276892 (+ 220us) server_negotiation.cc:407] Connection header received
0926 03:58:47.277519 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:47.277520 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:47.277533 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:47.277567 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:47.278568 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:47.279313 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:47.281040 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:47.281329 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:47.281371 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:47.285443 (+ 4072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:47.285445 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:47.285446 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:47.285454 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:47.287771 (+ 2317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:47.288383 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:47.288384 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:47.288384 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:47.288450 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:47.288687 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:47.288688 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:47.288688 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:47.288816 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:47.288870 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:47.289458 (+ 588us) server_negotiation.cc:299] Negotiation successful
0926 03:58:47.289535 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 03:58:47.290179 7608 heartbeater.cc:347] Connected to a master server at 127.0.98.61:35627
I20260926 03:58:47.290268 7608 heartbeater.cc:464] Registering TS with master...
I20260926 03:58:47.290441 7608 heartbeater.cc:511] Master 127.0.98.61:35627 requested a full tablet report, sending...
I20260926 03:58:47.290637 7606 heartbeater.cc:347] Connected to a master server at 127.0.98.60:34077
I20260926 03:58:47.290663 7606 heartbeater.cc:464] Registering TS with master...
I20260926 03:58:47.290707 7606 heartbeater.cc:511] Master 127.0.98.60:34077 requested a full tablet report, sending...
I20260926 03:58:47.290896 7102 ts_manager.cc:194] Registered new tserver with Master: 1e41eabd06d44a9ebab5ba2a76d5ea66 (127.0.98.2:43115)
I20260926 03:58:47.290953 7290 ts_manager.cc:194] Registered new tserver with Master: 1e41eabd06d44a9ebab5ba2a76d5ea66 (127.0.98.2:43115)
I20260926 03:58:47.291575 7611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:47.276551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41421 (local address 127.0.98.2:55779)
0926 03:58:47.276751 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:47.276799 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:47.277382 (+ 583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:47.277657 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:47.277658 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:47.278236 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:47.278451 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:47.278463 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:47.279410 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:47.279411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:47.280933 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:47.280936 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:47.281468 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:47.281468 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:47.281602 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:47.282338 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:47.282344 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:47.285346 (+ 3002us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:58:47.288070 (+ 2724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:47.288070 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:47.288071 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:47.288301 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:47.288518 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:47.288519 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:47.288519 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:47.288575 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:47.288903 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:58:47.288904 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:58:47.289310 (+ 406us) client_negotiation.cc:769] Sending connection context
0926 03:58:47.291394 (+ 2084us) client_negotiation.cc:240] Negotiation successful
0926 03:58:47.291509 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":296,"spinlock_wait_cycles":7040,"thread_start_us":141,"threads_started":1}
I20260926 03:58:47.291985 7607 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41421
I20260926 03:58:47.292049 7607 heartbeater.cc:464] Registering TS with master...
I20260926 03:58:47.292119 7607 heartbeater.cc:511] Master 127.0.98.62:41421 requested a full tablet report, sending...
I20260926 03:58:47.292315 6498 ts_manager.cc:194] Re-registered known tserver with Master: 1e41eabd06d44a9ebab5ba2a76d5ea66 (127.0.98.2:43115)
I20260926 03:58:47.292775 6498 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55779
W20260926 03:58:47.370883 7613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:58:47.371111 7613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:58:47.371135 7613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:58:47.372354 7613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:58:47.372402 7613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:58:47.372430 7613 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 03:58:47.373930 7613 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:43813
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=38727
--tserver_master_addrs=127.0.98.62:41421,127.0.98.61:35627,127.0.98.60:34077
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7613
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:58:47.374181 7613 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:58:47.374421 7613 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:58:47.379441 7619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:47.379441 7621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:47.379431 7618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:47.380085 7613 server_base.cc:1034] running on GCE node
I20260926 03:58:47.380268 7613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:58:47.380473 7613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:58:47.381632 7613 hybrid_clock.cc:648] HybridClock initialized: now 1790395127381622 us; error 46 us; skew 500 ppm
Sep 26 03:58:47 dist-test-slave-nhpw krb5kdc[6453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395127, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:58:47.384037 7613 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260926 03:58:47.384503 7613 webserver.cc:460] Webserver started at http://127.0.98.3:38727/ using document root <none> and password file <none>
I20260926 03:58:47.384778 7613 fs_manager.cc:359] Metadata directory not provided
I20260926 03:58:47.384857 7613 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:58:47.386234 7613 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 03:58:47.386837 7628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:47.387007 7613 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 03:58:47.387120 7613 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-2/wal
uuid: "241844d2ef2b4666bd8513fdc36a4efb"
format_stamp: "Formatted at 2026-09-26 03:58:46 on dist-test-slave-nhpw"
server_key: "79bc62c6f226e2e477d3e5764af86792"
server_key_iv: "9b66552daaf9270ed007ae462e87d421"
server_key_version: "encryptionkey@0"
I20260926 03:58:47.387463 7613 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:58:47.396219 7613 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:58:47.398393 7613 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:58:47.398564 7613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:58:47.398852 7613 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:58:47.399173 7613 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:58:47.399206 7613 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:47.399228 7613 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:58:47.399274 7613 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:47.405272 7613 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43813
I20260926 03:58:47.405320 7741 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43813 every 8 connection(s)
I20260926 03:58:47.405639 7613 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
I20260926 03:58:47.413156 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 7613
Sep 26 03:58:47 dist-test-slave-nhpw krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395127, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 26 03:58:47 dist-test-slave-nhpw krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395127, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 26 03:58:47 dist-test-slave-nhpw krb5kdc[6453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395127, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 03:58:47.418795 7153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:47.406600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39801 (local address 127.0.98.61:35627)
0926 03:58:47.406643 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:47.406645 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:47.406653 (+ 8us) server_negotiation.cc:407] Connection header received
0926 03:58:47.407739 (+ 1086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:47.407740 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:47.407752 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:47.407806 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:47.408425 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:47.409214 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:47.410655 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:47.410881 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:47.410974 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:47.415321 (+ 4347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:47.415323 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:47.415323 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:47.415331 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:47.417285 (+ 1954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:47.417702 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:47.417704 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:47.417704 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:47.417762 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:47.417988 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:47.417989 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:47.417990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:47.418147 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:47.418198 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:47.418598 (+ 400us) server_negotiation.cc:299] Negotiation successful
0926 03:58:47.418688 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 03:58:47.418910 7746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:47.406343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34077 (local address 127.0.98.3:47101)
0926 03:58:47.406518 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:47.406563 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:47.407667 (+ 1104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:47.407938 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:47.407939 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:47.408303 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:47.408386 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:47.408388 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:47.409355 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:47.409356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:47.410568 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:47.410570 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:47.411127 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:47.411128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:47.411186 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:47.411811 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:47.411815 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:47.414633 (+ 2818us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:58:47.417066 (+ 2433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:47.417072 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:47.417074 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:47.417356 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:47.417566 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:47.417567 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:47.417568 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:47.417704 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:47.418271 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:58:47.418271 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:58:47.418420 (+ 149us) client_negotiation.cc:769] Sending connection context
0926 03:58:47.418522 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 03:58:47.418682 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":55,"thread_start_us":83,"threads_started":1}
I20260926 03:58:47.419279 7747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:47.406342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35627 (local address 127.0.98.3:39801)
0926 03:58:47.406533 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:47.406563 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:47.407624 (+ 1061us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:47.407888 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:47.407891 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:47.408147 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:47.408332 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:47.408335 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:47.409352 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:47.409353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:47.410563 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:47.410565 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:47.410982 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:47.410983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:47.411117 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:47.411901 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:47.411906 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:47.415211 (+ 3305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:58:47.417384 (+ 2173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:47.417385 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:47.417385 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:47.417604 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:47.417845 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:47.417845 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:47.417846 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:47.417908 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:47.418253 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:58:47.418259 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:58:47.418424 (+ 165us) client_negotiation.cc:769] Sending connection context
0926 03:58:47.419065 (+ 641us) client_negotiation.cc:240] Negotiation successful
0926 03:58:47.419167 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":71,"thread_start_us":124,"threads_started":1}
I20260926 03:58:47.419724 7476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:47.406369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47101 (local address 127.0.98.60:34077)
0926 03:58:47.406423 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:47.406425 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:47.406640 (+ 215us) server_negotiation.cc:407] Connection header received
0926 03:58:47.407788 (+ 1148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:47.407790 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:47.407804 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:47.407841 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:47.408503 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:47.409227 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:47.410675 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:47.410968 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:47.411008 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:47.414757 (+ 3749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:47.414759 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:47.414759 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:47.414767 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:47.416984 (+ 2217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:47.417431 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:47.417432 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:47.417433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:47.417486 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:47.417778 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:47.417779 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:47.417779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:47.417897 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:47.417967 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:47.419524 (+ 1557us) server_negotiation.cc:299] Negotiation successful
0926 03:58:47.419606 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 03:58:47.420053 7744 heartbeater.cc:347] Connected to a master server at 127.0.98.61:35627
I20260926 03:58:47.420163 7744 heartbeater.cc:464] Registering TS with master...
I20260926 03:58:47.420284 6537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:47.406761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37801 (local address 127.0.98.62:41421)
0926 03:58:47.406834 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:47.406836 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:47.406846 (+ 10us) server_negotiation.cc:407] Connection header received
0926 03:58:47.407789 (+ 943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:47.407790 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:47.407804 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:47.407852 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:47.408716 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:47.409420 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:47.410674 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:47.410967 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:47.411008 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:47.415384 (+ 4376us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:47.415386 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:47.415386 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:47.415394 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:47.417927 (+ 2533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:47.419204 (+ 1277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:47.419205 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:47.419206 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:47.419268 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:47.419496 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:47.419497 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:47.419498 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:47.419650 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:47.419723 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:47.420117 (+ 394us) server_negotiation.cc:299] Negotiation successful
0926 03:58:47.420197 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 03:58:47.420354 7744 heartbeater.cc:511] Master 127.0.98.61:35627 requested a full tablet report, sending...
I20260926 03:58:47.420357 7748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:47.406620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41421 (local address 127.0.98.3:37801)
0926 03:58:47.406770 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:47.406778 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:47.407662 (+ 884us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:47.407938 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:47.407939 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:47.408305 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:47.408394 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:47.408395 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:47.409508 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:47.409508 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:47.410567 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:47.410570 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:47.411085 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:47.411086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:47.411152 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:47.411780 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:47.411791 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:47.415283 (+ 3492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:58:47.418002 (+ 2719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:47.418003 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:47.418003 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:47.418200 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:47.419343 (+ 1143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:47.419344 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:47.419344 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:47.419410 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:47.419754 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:58:47.419755 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:58:47.419801 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 03:58:47.419867 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 03:58:47.419979 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":81,"thread_start_us":63,"threads_started":1}
I20260926 03:58:47.420811 7743 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41421
I20260926 03:58:47.420869 7743 heartbeater.cc:464] Registering TS with master...
I20260926 03:58:47.420941 7743 heartbeater.cc:511] Master 127.0.98.62:41421 requested a full tablet report, sending...
I20260926 03:58:47.421173 7742 heartbeater.cc:347] Connected to a master server at 127.0.98.60:34077
I20260926 03:58:47.421219 7742 heartbeater.cc:464] Registering TS with master...
I20260926 03:58:47.421200 6498 ts_manager.cc:194] Re-registered known tserver with Master: 241844d2ef2b4666bd8513fdc36a4efb (127.0.98.3:43813)
I20260926 03:58:47.421275 7742 heartbeater.cc:511] Master 127.0.98.60:34077 requested a full tablet report, sending...
I20260926 03:58:47.421260 7102 ts_manager.cc:194] Registered new tserver with Master: 241844d2ef2b4666bd8513fdc36a4efb (127.0.98.3:43813)
I20260926 03:58:47.421514 7290 ts_manager.cc:194] Registered new tserver with Master: 241844d2ef2b4666bd8513fdc36a4efb (127.0.98.3:43813)
I20260926 03:58:47.421693 6498 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37801
I20260926 03:58:47.426242 7153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:47.414738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47082 (local address 127.0.98.61:35627)
0926 03:58:47.418891 (+ 4153us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:47.418893 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:47.418902 (+ 9us) server_negotiation.cc:407] Connection header received
0926 03:58:47.418925 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:47.418926 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:47.418938 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:47.419035 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:47.419762 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:47.420452 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:47.421602 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:47.421876 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:47.421910 (+ 34us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:47.423197 (+ 1287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:47.423199 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:47.423199 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:47.423205 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:47.424881 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:47.425294 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:47.425295 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:47.425295 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:47.425350 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:47.425544 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:47.425545 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:47.425546 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:47.425685 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:47.425755 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:47.425980 (+ 225us) server_negotiation.cc:299] Negotiation successful
0926 03:58:47.426080 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4136,"thread_start_us":51,"threads_started":1}
I20260926 03:58:47.427433 7757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:47.414780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45422 (local address 127.0.98.60:34077)
0926 03:58:47.415319 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:47.415321 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:47.415768 (+ 447us) server_negotiation.cc:407] Connection header received
0926 03:58:47.415790 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:47.415791 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:47.415803 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:47.415856 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:47.416730 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:47.417484 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:47.418552 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:47.418826 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:47.418867 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:47.424290 (+ 5423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:47.424292 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:47.424292 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:47.424299 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:47.426195 (+ 1896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:47.426654 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:47.426655 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:47.426656 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:47.426714 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:47.426909 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:47.426910 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:47.426911 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:47.427049 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:47.427136 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:47.427241 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 03:58:47.427315 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":48,"threads_started":1}
I20260926 03:58:47.429075 7754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:47.414396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47580 (local address 127.0.98.62:41421)
0926 03:58:47.415067 (+ 671us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:47.415069 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:47.415459 (+ 390us) server_negotiation.cc:407] Connection header received
0926 03:58:47.415516 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:47.415525 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:47.415540 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:47.418063 (+ 2523us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:47.421614 (+ 3551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:47.422374 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:47.423515 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:47.423901 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:47.423976 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:47.425558 (+ 1582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:47.425559 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:47.425559 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:47.425566 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:47.427839 (+ 2273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:47.428218 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:47.428219 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:47.428219 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:47.428281 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:47.428494 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:47.428496 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:47.428496 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:47.428639 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:47.428799 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:47.428884 (+ 85us) server_negotiation.cc:299] Negotiation successful
0926 03:58:47.428966 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":97,"threads_started":1}
I20260926 03:58:47.430564 6498 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47580:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:58:47.432013 6498 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:58:47.439531 7763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:47.435410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43813 (local address 127.0.0.1:37036)
0926 03:58:47.435713 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:47.435721 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:47.435760 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:47.436291 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:47.436292 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:47.436294 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:58:47.436349 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:47.436351 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:47.437324 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:47.437325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:47.438341 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:47.438343 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:47.439283 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:47.439284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:47.439345 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:47.439351 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 03:58:47.439411 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 03:58:47.439453 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":5,"thread_start_us":104,"threads_started":1}
I20260926 03:58:47.439543 7767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:47.435911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37036 (local address 127.0.98.3:43813)
0926 03:58:47.436101 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:47.436105 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:47.436114 (+ 9us) server_negotiation.cc:407] Connection header received
0926 03:58:47.436146 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:47.436148 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:47.436160 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:47.436289 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:58:47.436414 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:47.437216 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:47.438435 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:47.439177 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:47.439281 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:47.439369 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:47.439426 (+ 57us) server_negotiation.cc:299] Negotiation successful
0926 03:58:47.439461 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":92,"threads_started":1}
I20260926 03:58:47.439841 7056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:47.435380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37187 (local address 127.0.0.1:35746)
0926 03:58:47.435421 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:47.435432 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:47.435466 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:47.436157 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:47.436159 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:47.436160 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:58:47.436204 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:47.436206 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:47.437523 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:47.437524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:47.438670 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:47.438673 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:47.439621 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:47.439622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:47.439683 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:47.439686 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 03:58:47.439715 (+ 29us) client_negotiation.cc:240] Negotiation successful
0926 03:58:47.439756 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":1}
I20260926 03:58:47.439945 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:47.435410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43115 (local address 127.0.0.1:60650)
0926 03:58:47.435569 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:47.435579 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:47.435637 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:47.436322 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:47.436323 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:47.436325 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:58:47.436372 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:47.436373 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:47.437619 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:47.437620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:47.438711 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:47.438713 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:47.439661 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:47.439661 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:47.439715 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:47.439719 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 03:58:47.439825 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 03:58:47.439861 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":103,"threads_started":1}
I20260926 03:58:47.440215 7766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:47.435754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35746 (local address 127.0.98.1:37187)
0926 03:58:47.435905 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:47.435910 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:47.435922 (+ 12us) server_negotiation.cc:407] Connection header received
0926 03:58:47.436016 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:47.436018 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:47.436036 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:47.436273 (+ 237us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:58:47.436324 (+ 51us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:47.437427 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:47.438778 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:47.439515 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:47.440006 (+ 491us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:47.440083 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:47.440108 (+ 25us) server_negotiation.cc:299] Negotiation successful
0926 03:58:47.440145 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":75,"threads_started":1}
I20260926 03:58:47.440418 7676 tablet_service.cc:1467] Processing CreateTablet for tablet 0e7455428da24f15ad192e9b29f42fba (DEFAULT_TABLE table=test-table [id=651f00071f824528b0747bf898e9b716]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:58:47.440776 7676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e7455428da24f15ad192e9b29f42fba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:58:47.441048 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:47.435657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60650 (local address 127.0.98.2:43115)
0926 03:58:47.436114 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:47.436118 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:47.436128 (+ 10us) server_negotiation.cc:407] Connection header received
0926 03:58:47.436164 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:47.436166 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:47.436180 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:47.436256 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:58:47.436461 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:47.437537 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:47.438823 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:47.439520 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:47.440809 (+ 1289us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:47.440891 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:47.440918 (+ 27us) server_negotiation.cc:299] Negotiation successful
0926 03:58:47.440977 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":420,"thread_start_us":78,"threads_started":1}
I20260926 03:58:47.441643 7540 tablet_service.cc:1467] Processing CreateTablet for tablet 0e7455428da24f15ad192e9b29f42fba (DEFAULT_TABLE table=test-table [id=651f00071f824528b0747bf898e9b716]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:58:47.442037 7540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e7455428da24f15ad192e9b29f42fba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:58:47.444025 7768 tablet_bootstrap.cc:492] T 0e7455428da24f15ad192e9b29f42fba P 241844d2ef2b4666bd8513fdc36a4efb: Bootstrap starting.
I20260926 03:58:47.444911 7403 tablet_service.cc:1467] Processing CreateTablet for tablet 0e7455428da24f15ad192e9b29f42fba (DEFAULT_TABLE table=test-table [id=651f00071f824528b0747bf898e9b716]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:58:47.445154 7768 tablet_bootstrap.cc:654] T 0e7455428da24f15ad192e9b29f42fba P 241844d2ef2b4666bd8513fdc36a4efb: Neither blocks nor log segments found. Creating new log.
I20260926 03:58:47.445236 7403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e7455428da24f15ad192e9b29f42fba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:58:47.445454 7768 log.cc:824] T 0e7455428da24f15ad192e9b29f42fba P 241844d2ef2b4666bd8513fdc36a4efb: Log is configured to *not* fsync() on all Append() calls
I20260926 03:58:47.446331 7768 tablet_bootstrap.cc:492] T 0e7455428da24f15ad192e9b29f42fba P 241844d2ef2b4666bd8513fdc36a4efb: No bootstrap required, opened a new log
I20260926 03:58:47.446405 7768 ts_tablet_manager.cc:1397] T 0e7455428da24f15ad192e9b29f42fba P 241844d2ef2b4666bd8513fdc36a4efb: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260926 03:58:47.448611 7771 tablet_bootstrap.cc:492] T 0e7455428da24f15ad192e9b29f42fba P 226f08ac9eb34274ae439b88cea9050c: Bootstrap starting.
I20260926 03:58:47.448712 7769 tablet_bootstrap.cc:492] T 0e7455428da24f15ad192e9b29f42fba P 1e41eabd06d44a9ebab5ba2a76d5ea66: Bootstrap starting.
I20260926 03:58:47.448918 7768 raft_consensus.cc:348] T 0e7455428da24f15ad192e9b29f42fba P 241844d2ef2b4666bd8513fdc36a4efb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e41eabd06d44a9ebab5ba2a76d5ea66" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43115 } } peers { permanent_uuid: "241844d2ef2b4666bd8513fdc36a4efb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43813 } } peers { permanent_uuid: "226f08ac9eb34274ae439b88cea9050c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37187 } }
I20260926 03:58:47.449110 7768 raft_consensus.cc:374] T 0e7455428da24f15ad192e9b29f42fba P 241844d2ef2b4666bd8513fdc36a4efb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:58:47.449146 7768 raft_consensus.cc:729] T 0e7455428da24f15ad192e9b29f42fba P 241844d2ef2b4666bd8513fdc36a4efb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 241844d2ef2b4666bd8513fdc36a4efb, State: Initialized, Role: FOLLOWER
I20260926 03:58:47.449285 7771 tablet_bootstrap.cc:654] T 0e7455428da24f15ad192e9b29f42fba P 226f08ac9eb34274ae439b88cea9050c: Neither blocks nor log segments found. Creating new log.
I20260926 03:58:47.449316 7769 tablet_bootstrap.cc:654] T 0e7455428da24f15ad192e9b29f42fba P 1e41eabd06d44a9ebab5ba2a76d5ea66: Neither blocks nor log segments found. Creating new log.
I20260926 03:58:47.449278 7768 consensus_queue.cc:260] T 0e7455428da24f15ad192e9b29f42fba P 241844d2ef2b4666bd8513fdc36a4efb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e41eabd06d44a9ebab5ba2a76d5ea66" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43115 } } peers { permanent_uuid: "241844d2ef2b4666bd8513fdc36a4efb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43813 } } peers { permanent_uuid: "226f08ac9eb34274ae439b88cea9050c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37187 } }
I20260926 03:58:47.449543 7768 ts_tablet_manager.cc:1428] T 0e7455428da24f15ad192e9b29f42fba P 241844d2ef2b4666bd8513fdc36a4efb: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 03:58:47.449584 7743 heartbeater.cc:503] Master 127.0.98.62:41421 was elected leader, sending a full tablet report...
I20260926 03:58:47.449643 7771 log.cc:824] T 0e7455428da24f15ad192e9b29f42fba P 226f08ac9eb34274ae439b88cea9050c: Log is configured to *not* fsync() on all Append() calls
I20260926 03:58:47.449991 7769 log.cc:824] T 0e7455428da24f15ad192e9b29f42fba P 1e41eabd06d44a9ebab5ba2a76d5ea66: Log is configured to *not* fsync() on all Append() calls
I20260926 03:58:47.450942 7769 tablet_bootstrap.cc:492] T 0e7455428da24f15ad192e9b29f42fba P 1e41eabd06d44a9ebab5ba2a76d5ea66: No bootstrap required, opened a new log
I20260926 03:58:47.451035 7769 ts_tablet_manager.cc:1397] T 0e7455428da24f15ad192e9b29f42fba P 1e41eabd06d44a9ebab5ba2a76d5ea66: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 03:58:47.452328 7769 raft_consensus.cc:348] T 0e7455428da24f15ad192e9b29f42fba P 1e41eabd06d44a9ebab5ba2a76d5ea66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e41eabd06d44a9ebab5ba2a76d5ea66" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43115 } } peers { permanent_uuid: "241844d2ef2b4666bd8513fdc36a4efb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43813 } } peers { permanent_uuid: "226f08ac9eb34274ae439b88cea9050c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37187 } }
I20260926 03:58:47.452486 7769 raft_consensus.cc:374] T 0e7455428da24f15ad192e9b29f42fba P 1e41eabd06d44a9ebab5ba2a76d5ea66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:58:47.452531 7769 raft_consensus.cc:729] T 0e7455428da24f15ad192e9b29f42fba P 1e41eabd06d44a9ebab5ba2a76d5ea66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e41eabd06d44a9ebab5ba2a76d5ea66, State: Initialized, Role: FOLLOWER
I20260926 03:58:47.452673 7769 consensus_queue.cc:260] T 0e7455428da24f15ad192e9b29f42fba P 1e41eabd06d44a9ebab5ba2a76d5ea66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e41eabd06d44a9ebab5ba2a76d5ea66" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43115 } } peers { permanent_uuid: "241844d2ef2b4666bd8513fdc36a4efb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43813 } } peers { permanent_uuid: "226f08ac9eb34274ae439b88cea9050c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37187 } }
I20260926 03:58:47.452935 7769 ts_tablet_manager.cc:1428] T 0e7455428da24f15ad192e9b29f42fba P 1e41eabd06d44a9ebab5ba2a76d5ea66: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 03:58:47.453241 7771 tablet_bootstrap.cc:492] T 0e7455428da24f15ad192e9b29f42fba P 226f08ac9eb34274ae439b88cea9050c: No bootstrap required, opened a new log
I20260926 03:58:47.453326 7771 ts_tablet_manager.cc:1397] T 0e7455428da24f15ad192e9b29f42fba P 226f08ac9eb34274ae439b88cea9050c: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260926 03:58:47.455565 7771 raft_consensus.cc:348] T 0e7455428da24f15ad192e9b29f42fba P 226f08ac9eb34274ae439b88cea9050c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e41eabd06d44a9ebab5ba2a76d5ea66" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43115 } } peers { permanent_uuid: "241844d2ef2b4666bd8513fdc36a4efb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43813 } } peers { permanent_uuid: "226f08ac9eb34274ae439b88cea9050c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37187 } }
I20260926 03:58:47.455687 7771 raft_consensus.cc:374] T 0e7455428da24f15ad192e9b29f42fba P 226f08ac9eb34274ae439b88cea9050c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:58:47.455734 7771 raft_consensus.cc:729] T 0e7455428da24f15ad192e9b29f42fba P 226f08ac9eb34274ae439b88cea9050c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 226f08ac9eb34274ae439b88cea9050c, State: Initialized, Role: FOLLOWER
I20260926 03:58:47.455848 7771 consensus_queue.cc:260] T 0e7455428da24f15ad192e9b29f42fba P 226f08ac9eb34274ae439b88cea9050c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e41eabd06d44a9ebab5ba2a76d5ea66" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43115 } } peers { permanent_uuid: "241844d2ef2b4666bd8513fdc36a4efb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43813 } } peers { permanent_uuid: "226f08ac9eb34274ae439b88cea9050c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37187 } }
I20260926 03:58:47.456135 7771 ts_tablet_manager.cc:1428] T 0e7455428da24f15ad192e9b29f42fba P 226f08ac9eb34274ae439b88cea9050c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 03:58:47.456172 7470 heartbeater.cc:503] Master 127.0.98.62:41421 was elected leader, sending a full tablet report...
I20260926 03:58:47.456284 7607 heartbeater.cc:503] Master 127.0.98.62:41421 was elected leader, sending a full tablet report...
I20260926 03:58:47.480051 7757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:47.477570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45426 (local address 127.0.98.60:34077)
0926 03:58:47.477630 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:47.477633 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:47.477643 (+ 10us) server_negotiation.cc:407] Connection header received
0926 03:58:47.477671 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:47.477672 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:47.477698 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:47.477774 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:58:47.477910 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:47.478471 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:47.479346 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:47.479730 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:47.479810 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:47.479885 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:47.479944 (+ 59us) server_negotiation.cc:299] Negotiation successful
0926 03:58:47.479981 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 03:58:47.480054 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:47.477451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34077 (local address 127.0.0.1:45426)
0926 03:58:47.477515 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:47.477527 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:47.477576 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:47.477777 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:47.477778 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:47.477779 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:58:47.477837 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:47.477839 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:47.478540 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:47.478541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:47.479249 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:47.479251 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:47.479820 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:47.479821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:47.479878 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:47.479881 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 03:58:47.479938 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 03:58:47.479984 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 03:58:47.480440 7300 raft_consensus.cc:1203] T 00000000000000000000000000000000 P e84368bd28c54795852888b3d9163a75 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260926 03:58:47.481351 7329 sys_catalog.cc:455] T 00000000000000000000000000000000 P e84368bd28c54795852888b3d9163a75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0aeea5ec3a114f8c94fd2d2012fe312b. Latest consensus state: current_term: 1 leader_uuid: "0aeea5ec3a114f8c94fd2d2012fe312b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0aeea5ec3a114f8c94fd2d2012fe312b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41421 } } peers { permanent_uuid: "3f6ed0be592b4c6da0be1ae5feb24ff1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35627 } attrs { promote: false } } peers { permanent_uuid: "e84368bd28c54795852888b3d9163a75" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34077 } attrs { promote: true } } }
I20260926 03:58:47.481472 7329 sys_catalog.cc:458] T 00000000000000000000000000000000 P e84368bd28c54795852888b3d9163a75 [sys.catalog]: This master's current role is: LEARNER
W20260926 03:58:47.526800 7609 tablet.cc:2369] T 0e7455428da24f15ad192e9b29f42fba P 1e41eabd06d44a9ebab5ba2a76d5ea66: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:58:47.541395 7775 raft_consensus.cc:482] T 0e7455428da24f15ad192e9b29f42fba P 1e41eabd06d44a9ebab5ba2a76d5ea66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:58:47.541582 7775 raft_consensus.cc:504] T 0e7455428da24f15ad192e9b29f42fba P 1e41eabd06d44a9ebab5ba2a76d5ea66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e41eabd06d44a9ebab5ba2a76d5ea66" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43115 } } peers { permanent_uuid: "241844d2ef2b4666bd8513fdc36a4efb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43813 } } peers { permanent_uuid: "226f08ac9eb34274ae439b88cea9050c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37187 } }
I20260926 03:58:47.542130 7775 leader_election.cc:288] T 0e7455428da24f15ad192e9b29f42fba P 1e41eabd06d44a9ebab5ba2a76d5ea66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 241844d2ef2b4666bd8513fdc36a4efb (127.0.98.3:43813), 226f08ac9eb34274ae439b88cea9050c (127.0.98.1:37187)
I20260926 03:58:47.546077 7611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:47.542291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43813 (local address 127.0.98.2:37063)
0926 03:58:47.542354 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:47.542366 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:47.542418 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:47.542613 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:47.542614 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:47.542616 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:58:47.542693 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:47.542695 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:47.543660 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:47.543660 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:47.545000 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:47.545003 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:47.545799 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:47.545800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:47.545887 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:47.545890 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 03:58:47.545939 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 03:58:47.545976 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":68}
I20260926 03:58:47.546149 7767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:47.542359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37063 (local address 127.0.98.3:43813)
0926 03:58:47.542405 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:47.542407 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:47.542418 (+ 11us) server_negotiation.cc:407] Connection header received
0926 03:58:47.542505 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:47.542506 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:47.542519 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:47.542580 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:58:47.542802 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:47.543593 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:47.545125 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:47.545709 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:47.545791 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:47.545804 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:47.545966 (+ 162us) server_negotiation.cc:299] Negotiation successful
0926 03:58:47.546037 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 03:58:47.546233 7610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:47.542383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37187 (local address 127.0.98.2:60863)
0926 03:58:47.542447 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:47.542457 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:47.542490 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:47.542663 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:47.542664 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:47.542666 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:58:47.542710 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:47.542711 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:47.543657 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:47.543658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:47.545002 (+ 1344us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:47.545003 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:47.545857 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:47.545858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:47.545977 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:47.545980 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 03:58:47.546044 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 03:58:47.546086 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52,"mutex_wait_us":32}
I20260926 03:58:47.546264 7766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:47.542475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60863 (local address 127.0.98.1:37187)
0926 03:58:47.542517 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:47.542519 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:47.542527 (+ 8us) server_negotiation.cc:407] Connection header received
0926 03:58:47.542551 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:47.542551 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:47.542563 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:47.542657 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:58:47.542790 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:47.543550 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:47.545125 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:47.545752 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:47.545814 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:47.545822 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:47.546138 (+ 316us) server_negotiation.cc:299] Negotiation successful
0926 03:58:47.546189 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 03:58:47.546463 7696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e7455428da24f15ad192e9b29f42fba" candidate_uuid: "1e41eabd06d44a9ebab5ba2a76d5ea66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "241844d2ef2b4666bd8513fdc36a4efb" is_pre_election: true
I20260926 03:58:47.546607 7696 raft_consensus.cc:2454] T 0e7455428da24f15ad192e9b29f42fba P 241844d2ef2b4666bd8513fdc36a4efb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e41eabd06d44a9ebab5ba2a76d5ea66 in term 0.
I20260926 03:58:47.546593 7423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e7455428da24f15ad192e9b29f42fba" candidate_uuid: "1e41eabd06d44a9ebab5ba2a76d5ea66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "226f08ac9eb34274ae439b88cea9050c" is_pre_election: true
I20260926 03:58:47.546744 7423 raft_consensus.cc:2454] T 0e7455428da24f15ad192e9b29f42fba P 226f08ac9eb34274ae439b88cea9050c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e41eabd06d44a9ebab5ba2a76d5ea66 in term 0.
I20260926 03:58:47.547149 7495 leader_election.cc:302] T 0e7455428da24f15ad192e9b29f42fba P 1e41eabd06d44a9ebab5ba2a76d5ea66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e41eabd06d44a9ebab5ba2a76d5ea66, 241844d2ef2b4666bd8513fdc36a4efb; no voters:
I20260926 03:58:47.547312 7775 raft_consensus.cc:2781] T 0e7455428da24f15ad192e9b29f42fba P 1e41eabd06d44a9ebab5ba2a76d5ea66 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:58:47.547379 7775 raft_consensus.cc:482] T 0e7455428da24f15ad192e9b29f42fba P 1e41eabd06d44a9ebab5ba2a76d5ea66 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:58:47.547396 7775 raft_consensus.cc:3037] T 0e7455428da24f15ad192e9b29f42fba P 1e41eabd06d44a9ebab5ba2a76d5ea66 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:58:47.548084 7775 raft_consensus.cc:504] T 0e7455428da24f15ad192e9b29f42fba P 1e41eabd06d44a9ebab5ba2a76d5ea66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e41eabd06d44a9ebab5ba2a76d5ea66" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43115 } } peers { permanent_uuid: "241844d2ef2b4666bd8513fdc36a4efb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43813 } } peers { permanent_uuid: "226f08ac9eb34274ae439b88cea9050c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37187 } }
I20260926 03:58:47.548240 7775 leader_election.cc:288] T 0e7455428da24f15ad192e9b29f42fba P 1e41eabd06d44a9ebab5ba2a76d5ea66 [CANDIDATE]: Term 1 election: Requested vote from peers 241844d2ef2b4666bd8513fdc36a4efb (127.0.98.3:43813), 226f08ac9eb34274ae439b88cea9050c (127.0.98.1:37187)
I20260926 03:58:47.548425 7696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e7455428da24f15ad192e9b29f42fba" candidate_uuid: "1e41eabd06d44a9ebab5ba2a76d5ea66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "241844d2ef2b4666bd8513fdc36a4efb"
I20260926 03:58:47.548511 7696 raft_consensus.cc:3037] T 0e7455428da24f15ad192e9b29f42fba P 241844d2ef2b4666bd8513fdc36a4efb [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:58:47.548710 7423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e7455428da24f15ad192e9b29f42fba" candidate_uuid: "1e41eabd06d44a9ebab5ba2a76d5ea66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "226f08ac9eb34274ae439b88cea9050c"
I20260926 03:58:47.548822 7423 raft_consensus.cc:3037] T 0e7455428da24f15ad192e9b29f42fba P 226f08ac9eb34274ae439b88cea9050c [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:58:47.549340 7696 raft_consensus.cc:2454] T 0e7455428da24f15ad192e9b29f42fba P 241844d2ef2b4666bd8513fdc36a4efb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e41eabd06d44a9ebab5ba2a76d5ea66 in term 1.
I20260926 03:58:47.549508 7495 leader_election.cc:302] T 0e7455428da24f15ad192e9b29f42fba P 1e41eabd06d44a9ebab5ba2a76d5ea66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e41eabd06d44a9ebab5ba2a76d5ea66, 241844d2ef2b4666bd8513fdc36a4efb; no voters:
I20260926 03:58:47.549585 7775 raft_consensus.cc:2781] T 0e7455428da24f15ad192e9b29f42fba P 1e41eabd06d44a9ebab5ba2a76d5ea66 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:58:47.549610 7423 raft_consensus.cc:2454] T 0e7455428da24f15ad192e9b29f42fba P 226f08ac9eb34274ae439b88cea9050c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e41eabd06d44a9ebab5ba2a76d5ea66 in term 1.
I20260926 03:58:47.549731 7775 raft_consensus.cc:686] T 0e7455428da24f15ad192e9b29f42fba P 1e41eabd06d44a9ebab5ba2a76d5ea66 [term 1 LEADER]: Becoming Leader. State: Replica: 1e41eabd06d44a9ebab5ba2a76d5ea66, State: Running, Role: LEADER
I20260926 03:58:47.549826 7775 consensus_queue.cc:237] T 0e7455428da24f15ad192e9b29f42fba P 1e41eabd06d44a9ebab5ba2a76d5ea66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e41eabd06d44a9ebab5ba2a76d5ea66" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43115 } } peers { permanent_uuid: "241844d2ef2b4666bd8513fdc36a4efb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43813 } } peers { permanent_uuid: "226f08ac9eb34274ae439b88cea9050c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37187 } }
I20260926 03:58:47.550714 6497 catalog_manager.cc:5495] T 0e7455428da24f15ad192e9b29f42fba P 1e41eabd06d44a9ebab5ba2a76d5ea66 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e41eabd06d44a9ebab5ba2a76d5ea66 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "1e41eabd06d44a9ebab5ba2a76d5ea66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1e41eabd06d44a9ebab5ba2a76d5ea66" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "241844d2ef2b4666bd8513fdc36a4efb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "226f08ac9eb34274ae439b88cea9050c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37187 } health_report { overall_health: UNKNOWN } } }
I20260926 03:58:47.551362 7783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0aeea5ec3a114f8c94fd2d2012fe312b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0aeea5ec3a114f8c94fd2d2012fe312b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "0aeea5ec3a114f8c94fd2d2012fe312b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41421 } } peers { permanent_uuid: "3f6ed0be592b4c6da0be1ae5feb24ff1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35627 } attrs { promote: false } } peers { permanent_uuid: "e84368bd28c54795852888b3d9163a75" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 34077 } attrs { promote: true } } }
I20260926 03:58:47.551460 7783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0aeea5ec3a114f8c94fd2d2012fe312b [sys.catalog]: This master's current role is: LEADER
I20260926 03:58:47.552016 7057 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 0aeea5ec3a114f8c94fd2d2012fe312b: attempting to promote NON_VOTER e84368bd28c54795852888b3d9163a75 to VOTER
I20260926 03:58:47.552137 7057 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0aeea5ec3a114f8c94fd2d2012fe312b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0aeea5ec3a114f8c94fd2d2012fe312b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41421 } } peers { permanent_uuid: "3f6ed0be592b4c6da0be1ae5feb24ff1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35627 } attrs { promote: false } } peers { permanent_uuid: "e84368bd28c54795852888b3d9163a75" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34077 } attrs { promote: false } }
I20260926 03:58:47.552510 7112 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3f6ed0be592b4c6da0be1ae5feb24ff1 [term 1 FOLLOWER]: Refusing update from remote peer 0aeea5ec3a114f8c94fd2d2012fe312b: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260926 03:58:47.552517 7300 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e84368bd28c54795852888b3d9163a75 [term 1 LEARNER]: Refusing update from remote peer 0aeea5ec3a114f8c94fd2d2012fe312b: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260926 03:58:47.552747 7783 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0aeea5ec3a114f8c94fd2d2012fe312b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e84368bd28c54795852888b3d9163a75" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34077 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260926 03:58:47.552832 7057 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0aeea5ec3a114f8c94fd2d2012fe312b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f6ed0be592b4c6da0be1ae5feb24ff1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35627 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260926 03:58:47.553412 7057 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0aeea5ec3a114f8c94fd2d2012fe312b [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, e84368bd28c54795852888b3d9163a75 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0aeea5ec3a114f8c94fd2d2012fe312b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41421 } } peers { permanent_uuid: "3f6ed0be592b4c6da0be1ae5feb24ff1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35627 } attrs { promote: false } } peers { permanent_uuid: "e84368bd28c54795852888b3d9163a75" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34077 } attrs { promote: false } } }
I20260926 03:58:47.553637 7300 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e84368bd28c54795852888b3d9163a75 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, e84368bd28c54795852888b3d9163a75 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0aeea5ec3a114f8c94fd2d2012fe312b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41421 } } peers { permanent_uuid: "3f6ed0be592b4c6da0be1ae5feb24ff1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35627 } attrs { promote: false } } peers { permanent_uuid: "e84368bd28c54795852888b3d9163a75" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34077 } attrs { promote: false } } }
I20260926 03:58:47.554333 7057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0aeea5ec3a114f8c94fd2d2012fe312b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0aeea5ec3a114f8c94fd2d2012fe312b" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0aeea5ec3a114f8c94fd2d2012fe312b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41421 } } peers { permanent_uuid: "3f6ed0be592b4c6da0be1ae5feb24ff1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35627 } attrs { promote: false } } peers { permanent_uuid: "e84368bd28c54795852888b3d9163a75" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34077 } attrs { promote: false } } }
I20260926 03:58:47.554414 7057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0aeea5ec3a114f8c94fd2d2012fe312b [sys.catalog]: This master's current role is: LEADER
I20260926 03:58:47.554476 7057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0aeea5ec3a114f8c94fd2d2012fe312b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0aeea5ec3a114f8c94fd2d2012fe312b" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0aeea5ec3a114f8c94fd2d2012fe312b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41421 } } peers { permanent_uuid: "3f6ed0be592b4c6da0be1ae5feb24ff1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35627 } attrs { promote: false } } peers { permanent_uuid: "e84368bd28c54795852888b3d9163a75" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34077 } attrs { promote: false } } }
I20260926 03:58:47.554544 7250 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0aeea5ec3a114f8c94fd2d2012fe312b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0aeea5ec3a114f8c94fd2d2012fe312b" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0aeea5ec3a114f8c94fd2d2012fe312b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41421 } } peers { permanent_uuid: "3f6ed0be592b4c6da0be1ae5feb24ff1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35627 } attrs { promote: false } } peers { permanent_uuid: "e84368bd28c54795852888b3d9163a75" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34077 } attrs { promote: false } } }
I20260926 03:58:47.554678 7250 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0aeea5ec3a114f8c94fd2d2012fe312b [sys.catalog]: This master's current role is: LEADER
I20260926 03:58:47.554718 7329 sys_catalog.cc:455] T 00000000000000000000000000000000 P e84368bd28c54795852888b3d9163a75 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0aeea5ec3a114f8c94fd2d2012fe312b" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0aeea5ec3a114f8c94fd2d2012fe312b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41421 } } peers { permanent_uuid: "3f6ed0be592b4c6da0be1ae5feb24ff1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35627 } attrs { promote: false } } peers { permanent_uuid: "e84368bd28c54795852888b3d9163a75" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34077 } attrs { promote: false } } }
I20260926 03:58:47.554800 7329 sys_catalog.cc:458] T 00000000000000000000000000000000 P e84368bd28c54795852888b3d9163a75 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:58:47.554955 7112 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3f6ed0be592b4c6da0be1ae5feb24ff1 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, e84368bd28c54795852888b3d9163a75 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0aeea5ec3a114f8c94fd2d2012fe312b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41421 } } peers { permanent_uuid: "3f6ed0be592b4c6da0be1ae5feb24ff1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35627 } attrs { promote: false } } peers { permanent_uuid: "e84368bd28c54795852888b3d9163a75" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34077 } attrs { promote: false } } }
I20260926 03:58:47.554560 7057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0aeea5ec3a114f8c94fd2d2012fe312b [sys.catalog]: This master's current role is: LEADER
I20260926 03:58:47.555823 7785 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f6ed0be592b4c6da0be1ae5feb24ff1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0aeea5ec3a114f8c94fd2d2012fe312b" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0aeea5ec3a114f8c94fd2d2012fe312b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41421 } } peers { permanent_uuid: "3f6ed0be592b4c6da0be1ae5feb24ff1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35627 } attrs { promote: false } } peers { permanent_uuid: "e84368bd28c54795852888b3d9163a75" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34077 } attrs { promote: false } } }
I20260926 03:58:47.555981 7785 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f6ed0be592b4c6da0be1ae5feb24ff1 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:58:47.583218 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:47.580893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60666 (local address 127.0.98.2:43115)
0926 03:58:47.580983 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:47.580985 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:47.580994 (+ 9us) server_negotiation.cc:407] Connection header received
0926 03:58:47.581029 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:47.581029 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:47.581049 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:47.581126 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:58:47.581279 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:47.581749 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:47.582550 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:47.582738 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:47.582803 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:47.582932 (+ 129us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:58:47.583039 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:58:47.583101 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:47.583143 (+ 42us) server_negotiation.cc:299] Negotiation successful
0926 03:58:47.583160 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 03:58:47.585189 7696 raft_consensus.cc:1261] T 0e7455428da24f15ad192e9b29f42fba P 241844d2ef2b4666bd8513fdc36a4efb [term 1 FOLLOWER]: Refusing update from remote peer 1e41eabd06d44a9ebab5ba2a76d5ea66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:58:47.585278 7423 raft_consensus.cc:1261] T 0e7455428da24f15ad192e9b29f42fba P 226f08ac9eb34274ae439b88cea9050c [term 1 FOLLOWER]: Refusing update from remote peer 1e41eabd06d44a9ebab5ba2a76d5ea66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:58:47.585587 7781 consensus_queue.cc:1035] T 0e7455428da24f15ad192e9b29f42fba P 1e41eabd06d44a9ebab5ba2a76d5ea66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "241844d2ef2b4666bd8513fdc36a4efb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:58:47.585722 7775 consensus_queue.cc:1035] T 0e7455428da24f15ad192e9b29f42fba P 1e41eabd06d44a9ebab5ba2a76d5ea66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "226f08ac9eb34274ae439b88cea9050c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:58:47.589241 7788 mvcc.cc:205] Tried to move back new op lower bound from 7333458442585571328 to 7333458442444713984. Current Snapshot: MvccSnapshot[applied={T|T < 7333458442585571328}]
I20260926 03:58:47.590325 7789 mvcc.cc:205] Tried to move back new op lower bound from 7333458442585571328 to 7333458442444713984. Current Snapshot: MvccSnapshot[applied={T|T < 7333458442585571328}]
I20260926 03:58:47.591609 6497 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:58:47.591697 6497 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:58:47.592968 6497 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0aeea5ec3a114f8c94fd2d2012fe312b: Sending DeleteTablet for 3 replicas of tablet 0e7455428da24f15ad192e9b29f42fba
I20260926 03:58:47.593391 7403 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e7455428da24f15ad192e9b29f42fba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:58:47 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35746
I20260926 03:58:47.593655 7540 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e7455428da24f15ad192e9b29f42fba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:58:47 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60650
I20260926 03:58:47.593694 7796 tablet_replica.cc:330] T 0e7455428da24f15ad192e9b29f42fba P 226f08ac9eb34274ae439b88cea9050c: stopping tablet replica
I20260926 03:58:47.593838 7796 raft_consensus.cc:2229] T 0e7455428da24f15ad192e9b29f42fba P 226f08ac9eb34274ae439b88cea9050c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:58:47.593956 7796 raft_consensus.cc:2258] T 0e7455428da24f15ad192e9b29f42fba P 226f08ac9eb34274ae439b88cea9050c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:58:47.594030 7676 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e7455428da24f15ad192e9b29f42fba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:58:47 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37036
I20260926 03:58:47.594148 7798 tablet_replica.cc:330] T 0e7455428da24f15ad192e9b29f42fba P 1e41eabd06d44a9ebab5ba2a76d5ea66: stopping tablet replica
I20260926 03:58:47.594252 7798 raft_consensus.cc:2229] T 0e7455428da24f15ad192e9b29f42fba P 1e41eabd06d44a9ebab5ba2a76d5ea66 [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:58:47.594362 7798 raft_consensus.cc:2258] T 0e7455428da24f15ad192e9b29f42fba P 1e41eabd06d44a9ebab5ba2a76d5ea66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:58:47.594376 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 6466
I20260926 03:58:47.594863 7796 ts_tablet_manager.cc:1905] T 0e7455428da24f15ad192e9b29f42fba P 226f08ac9eb34274ae439b88cea9050c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:58:47.595813 7798 ts_tablet_manager.cc:1905] T 0e7455428da24f15ad192e9b29f42fba P 1e41eabd06d44a9ebab5ba2a76d5ea66: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:58:47.596840 7796 ts_tablet_manager.cc:1918] T 0e7455428da24f15ad192e9b29f42fba P 226f08ac9eb34274ae439b88cea9050c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:58:47.596930 7796 log.cc:1196] T 0e7455428da24f15ad192e9b29f42fba P 226f08ac9eb34274ae439b88cea9050c: Deleting WAL directory at /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-0/wal/wals/0e7455428da24f15ad192e9b29f42fba
I20260926 03:58:47.597208 7796 ts_tablet_manager.cc:1939] T 0e7455428da24f15ad192e9b29f42fba P 226f08ac9eb34274ae439b88cea9050c: Deleting consensus metadata
I20260926 03:58:47.597581 7798 ts_tablet_manager.cc:1918] T 0e7455428da24f15ad192e9b29f42fba P 1e41eabd06d44a9ebab5ba2a76d5ea66: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:58:47.597680 7798 log.cc:1196] T 0e7455428da24f15ad192e9b29f42fba P 1e41eabd06d44a9ebab5ba2a76d5ea66: Deleting WAL directory at /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-1/wal/wals/0e7455428da24f15ad192e9b29f42fba
I20260926 03:58:47.597957 7798 ts_tablet_manager.cc:1939] T 0e7455428da24f15ad192e9b29f42fba P 1e41eabd06d44a9ebab5ba2a76d5ea66: Deleting consensus metadata
W20260926 03:58:47.600688 7494 connection.cc:535] server connection from 127.0.0.1:60650 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 03:58:47.600871 7357 connection.cc:535] server connection from 127.0.0.1:35746 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 03:58:47.601209 7799 tablet_replica.cc:330] T 0e7455428da24f15ad192e9b29f42fba P 241844d2ef2b4666bd8513fdc36a4efb: stopping tablet replica
I20260926 03:58:47.601336 7799 raft_consensus.cc:2229] T 0e7455428da24f15ad192e9b29f42fba P 241844d2ef2b4666bd8513fdc36a4efb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:58:47.601468 7799 raft_consensus.cc:2258] T 0e7455428da24f15ad192e9b29f42fba P 241844d2ef2b4666bd8513fdc36a4efb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:58:47.601956 7799 ts_tablet_manager.cc:1905] T 0e7455428da24f15ad192e9b29f42fba P 241844d2ef2b4666bd8513fdc36a4efb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:58:47.604043 7799 ts_tablet_manager.cc:1918] T 0e7455428da24f15ad192e9b29f42fba P 241844d2ef2b4666bd8513fdc36a4efb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:58:47.604121 7799 log.cc:1196] T 0e7455428da24f15ad192e9b29f42fba P 241844d2ef2b4666bd8513fdc36a4efb: Deleting WAL directory at /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-2/wal/wals/0e7455428da24f15ad192e9b29f42fba
I20260926 03:58:47.604403 7799 ts_tablet_manager.cc:1939] T 0e7455428da24f15ad192e9b29f42fba P 241844d2ef2b4666bd8513fdc36a4efb: Deleting consensus metadata
W20260926 03:58:47.604802 7632 connection.cc:441] server connection from 127.0.0.1:37036 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:37036 (request call id 1) could send its response
I20260926 03:58:47.611053 7153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:47.602368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47092 (local address 127.0.98.61:35627)
0926 03:58:47.602415 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:47.602417 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:47.603767 (+ 1350us) server_negotiation.cc:407] Connection header received
0926 03:58:47.603831 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:47.603832 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:47.603843 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:47.603894 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:47.604656 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:47.605223 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:47.606199 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:47.606480 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:47.606544 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:47.607826 (+ 1282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:47.607827 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:47.607828 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:47.607834 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:47.609507 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:47.610229 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:47.610230 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:47.610231 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:47.610283 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:47.610509 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:47.610510 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:47.610510 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:47.610640 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:47.610737 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:47.610883 (+ 146us) server_negotiation.cc:299] Negotiation successful
0926 03:58:47.610959 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 03:58:47.611181 7476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:47.602169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45454 (local address 127.0.98.60:34077)
0926 03:58:47.602211 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:47.602213 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:47.602487 (+ 274us) server_negotiation.cc:407] Connection header received
0926 03:58:47.602597 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:47.602599 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:47.602618 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:47.602685 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:47.603443 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:47.604282 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:47.605475 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:47.605763 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:47.605827 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:47.607489 (+ 1662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:47.607491 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:47.607491 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:47.607498 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:47.609278 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:47.609645 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:47.609646 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:47.609647 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:47.609698 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:47.609912 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:47.609914 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:47.609914 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:47.610059 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:47.610153 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:47.610994 (+ 841us) server_negotiation.cc:299] Negotiation successful
0926 03:58:47.611067 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 03:58:47.611912 7757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:47.593714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45442 (local address 127.0.98.60:34077)
0926 03:58:47.593755 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:47.593757 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:47.604776 (+ 11019us) server_negotiation.cc:407] Connection header received
0926 03:58:47.604868 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:47.604869 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:47.604877 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:47.604905 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:47.605620 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:47.606300 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:47.607327 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:47.607635 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:47.607674 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:47.609122 (+ 1448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:47.609123 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:47.609124 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:47.609130 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:47.610735 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:47.611081 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:47.611082 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:47.611082 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:47.611127 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:47.611335 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:47.611337 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:47.611337 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:47.611480 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:47.611539 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:47.611743 (+ 204us) server_negotiation.cc:299] Negotiation successful
0926 03:58:47.611811 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 03:58:48.590085 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:48.589693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41421 (local address 127.0.98.1:53805)
0926 03:58:48.589873 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:48.589958 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":73,"threads_started":1}
W20260926 03:58:48.590401 7470 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41421 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:41421: connect: Connection refused (error 111)
I20260926 03:58:48.590740 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:48.590538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41421 (local address 127.0.98.2:41951)
0926 03:58:48.590650 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:48.590707 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":58,"threads_started":1}
W20260926 03:58:48.590945 7607 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41421 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:41421: connect: Connection refused (error 111)
I20260926 03:58:48.591586 7808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:48.591308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41421 (local address 127.0.98.3:47135)
0926 03:58:48.591467 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:48.591530 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":37,"threads_started":1}
W20260926 03:58:48.591773 7743 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41421 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:41421: connect: Connection refused (error 111)
I20260926 03:58:49.093604 7809 raft_consensus.cc:482] T 00000000000000000000000000000000 P e84368bd28c54795852888b3d9163a75 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0aeea5ec3a114f8c94fd2d2012fe312b)
I20260926 03:58:49.093710 7810 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f6ed0be592b4c6da0be1ae5feb24ff1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0aeea5ec3a114f8c94fd2d2012fe312b)
I20260926 03:58:49.093732 7809 raft_consensus.cc:504] T 00000000000000000000000000000000 P e84368bd28c54795852888b3d9163a75 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0aeea5ec3a114f8c94fd2d2012fe312b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41421 } } peers { permanent_uuid: "3f6ed0be592b4c6da0be1ae5feb24ff1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35627 } attrs { promote: false } } peers { permanent_uuid: "e84368bd28c54795852888b3d9163a75" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34077 } attrs { promote: false } }
I20260926 03:58:49.093870 7810 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f6ed0be592b4c6da0be1ae5feb24ff1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0aeea5ec3a114f8c94fd2d2012fe312b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41421 } } peers { permanent_uuid: "3f6ed0be592b4c6da0be1ae5feb24ff1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35627 } attrs { promote: false } } peers { permanent_uuid: "e84368bd28c54795852888b3d9163a75" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34077 } attrs { promote: false } }
I20260926 03:58:49.094173 7810 leader_election.cc:288] T 00000000000000000000000000000000 P 3f6ed0be592b4c6da0be1ae5feb24ff1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0aeea5ec3a114f8c94fd2d2012fe312b (127.0.98.62:41421), e84368bd28c54795852888b3d9163a75 (127.0.98.60:34077)
I20260926 03:58:49.094313 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:49.094113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41421 (local address 127.0.0.1:48048)
0926 03:58:49.094223 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:49.094269 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":68,"threads_started":1}
W20260926 03:58:49.094413 7276 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41421: connect: Connection refused (error 111)
I20260926 03:58:49.094427 7809 leader_election.cc:288] T 00000000000000000000000000000000 P e84368bd28c54795852888b3d9163a75 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0aeea5ec3a114f8c94fd2d2012fe312b (127.0.98.62:41421), 3f6ed0be592b4c6da0be1ae5feb24ff1 (127.0.98.61:35627)
I20260926 03:58:49.094632 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:49.094249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41421 (local address 127.0.0.1:48056)
0926 03:58:49.094540 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:49.094594 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":70,"threads_started":1}
W20260926 03:58:49.094789 7088 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41421: connect: Connection refused (error 111)
I20260926 03:58:49.094930 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:49.094773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41421 (local address 127.0.0.1:48058)
0926 03:58:49.094873 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:49.094897 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":64,"threads_started":1}
W20260926 03:58:49.095057 7276 leader_election.cc:334] T 00000000000000000000000000000000 P e84368bd28c54795852888b3d9163a75 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0aeea5ec3a114f8c94fd2d2012fe312b (127.0.98.62:41421): Network error: Client connection negotiation failed: client connection to 127.0.98.62:41421: connect: Connection refused (error 111)
I20260926 03:58:49.095255 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:49.095156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41421 (local address 127.0.0.1:48060)
0926 03:58:49.095192 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:49.095222 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260926 03:58:49.095343 7088 leader_election.cc:334] T 00000000000000000000000000000000 P 3f6ed0be592b4c6da0be1ae5feb24ff1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0aeea5ec3a114f8c94fd2d2012fe312b (127.0.98.62:41421): Network error: Client connection negotiation failed: client connection to 127.0.98.62:41421: connect: Connection refused (error 111)
I20260926 03:58:49.097963 7819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:49.094993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47106 (local address 127.0.98.61:35627)
0926 03:58:49.095132 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:49.095134 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:49.095143 (+ 9us) server_negotiation.cc:407] Connection header received
0926 03:58:49.095168 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:49.095169 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:49.095180 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:49.095275 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:58:49.095447 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:49.095995 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:49.097232 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:49.097662 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:49.097715 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:49.097724 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:49.097894 (+ 170us) server_negotiation.cc:299] Negotiation successful
0926 03:58:49.097921 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":78,"threads_started":1}
I20260926 03:58:49.097990 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:49.094627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:35627 (local address 127.0.0.1:47106)
0926 03:58:49.094750 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:49.094760 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:49.094809 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:49.095285 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:49.095286 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:49.095289 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:58:49.095351 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:49.095369 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:49.096082 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:49.096083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:49.097144 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:49.097146 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:49.097744 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:49.097745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:49.097806 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:49.097810 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 03:58:49.097874 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 03:58:49.097916 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":70,"threads_started":1}
I20260926 03:58:49.098248 7112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e84368bd28c54795852888b3d9163a75" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3f6ed0be592b4c6da0be1ae5feb24ff1" is_pre_election: true
I20260926 03:58:49.098349 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:49.094470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45460 (local address 127.0.98.60:34077)
0926 03:58:49.094646 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:49.094648 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:49.094658 (+ 10us) server_negotiation.cc:407] Connection header received
0926 03:58:49.095304 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:49.095305 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:49.095425 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:49.095486 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:58:49.095684 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:49.096384 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:49.097494 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:49.097978 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:49.098045 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:49.098054 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:49.098247 (+ 193us) server_negotiation.cc:299] Negotiation successful
0926 03:58:49.098286 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":45,"threads_started":1}
I20260926 03:58:49.098351 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:49.094321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34077 (local address 127.0.0.1:45460)
0926 03:58:49.094588 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:49.094602 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:49.094669 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:49.095549 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:49.095550 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:49.095551 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:58:49.095602 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:49.095604 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:49.096473 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:49.096474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:49.097348 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:49.097349 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:49.098091 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:49.098092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:49.098186 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:49.098191 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 03:58:49.098244 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 03:58:49.098283 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":1,"thread_start_us":57,"threads_started":1}
I20260926 03:58:49.098358 7112 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3f6ed0be592b4c6da0be1ae5feb24ff1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e84368bd28c54795852888b3d9163a75 in term 1.
I20260926 03:58:49.098519 7276 leader_election.cc:302] T 00000000000000000000000000000000 P e84368bd28c54795852888b3d9163a75 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 3f6ed0be592b4c6da0be1ae5feb24ff1, e84368bd28c54795852888b3d9163a75; no voters: 0aeea5ec3a114f8c94fd2d2012fe312b
I20260926 03:58:49.098649 7809 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e84368bd28c54795852888b3d9163a75 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 03:58:49.098694 7809 raft_consensus.cc:482] T 00000000000000000000000000000000 P e84368bd28c54795852888b3d9163a75 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0aeea5ec3a114f8c94fd2d2012fe312b)
I20260926 03:58:49.098719 7809 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e84368bd28c54795852888b3d9163a75 [term 1 FOLLOWER]: Advancing to term 2
I20260926 03:58:49.098708 7300 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3f6ed0be592b4c6da0be1ae5feb24ff1" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e84368bd28c54795852888b3d9163a75" is_pre_election: true
I20260926 03:58:49.099526 7809 raft_consensus.cc:504] T 00000000000000000000000000000000 P e84368bd28c54795852888b3d9163a75 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0aeea5ec3a114f8c94fd2d2012fe312b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41421 } } peers { permanent_uuid: "3f6ed0be592b4c6da0be1ae5feb24ff1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35627 } attrs { promote: false } } peers { permanent_uuid: "e84368bd28c54795852888b3d9163a75" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34077 } attrs { promote: false } }
I20260926 03:58:49.099674 7809 leader_election.cc:288] T 00000000000000000000000000000000 P e84368bd28c54795852888b3d9163a75 [CANDIDATE]: Term 2 election: Requested vote from peers 0aeea5ec3a114f8c94fd2d2012fe312b (127.0.98.62:41421), 3f6ed0be592b4c6da0be1ae5feb24ff1 (127.0.98.61:35627)
I20260926 03:58:49.099676 7300 raft_consensus.cc:2379] T 00000000000000000000000000000000 P e84368bd28c54795852888b3d9163a75 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3f6ed0be592b4c6da0be1ae5feb24ff1 in current term 2: Already voted for candidate e84368bd28c54795852888b3d9163a75 in this term.
I20260926 03:58:49.099862 7112 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e84368bd28c54795852888b3d9163a75" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "3f6ed0be592b4c6da0be1ae5feb24ff1"
I20260926 03:58:49.099953 7112 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f6ed0be592b4c6da0be1ae5feb24ff1 [term 1 FOLLOWER]: Advancing to term 2
I20260926 03:58:49.099974 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:49.099749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41421 (local address 127.0.0.1:48070)
0926 03:58:49.099915 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:49.099944 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
I20260926 03:58:49.099957 7089 leader_election.cc:302] T 00000000000000000000000000000000 P 3f6ed0be592b4c6da0be1ae5feb24ff1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3f6ed0be592b4c6da0be1ae5feb24ff1; no voters: 0aeea5ec3a114f8c94fd2d2012fe312b, e84368bd28c54795852888b3d9163a75
I20260926 03:58:49.100628 7112 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3f6ed0be592b4c6da0be1ae5feb24ff1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e84368bd28c54795852888b3d9163a75 in term 2.
I20260926 03:58:49.100764 7810 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 3f6ed0be592b4c6da0be1ae5feb24ff1 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 03:58:49.100818 7276 leader_election.cc:302] T 00000000000000000000000000000000 P e84368bd28c54795852888b3d9163a75 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f6ed0be592b4c6da0be1ae5feb24ff1, e84368bd28c54795852888b3d9163a75; no voters:
I20260926 03:58:49.100998 7809 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e84368bd28c54795852888b3d9163a75 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 03:58:49.101078 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:49.100962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41421 (local address 127.0.0.1:48084)
0926 03:58:49.101009 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:49.101053 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 03:58:49.101150 7809 raft_consensus.cc:686] T 00000000000000000000000000000000 P e84368bd28c54795852888b3d9163a75 [term 2 LEADER]: Becoming Leader. State: Replica: e84368bd28c54795852888b3d9163a75, State: Running, Role: LEADER
W20260926 03:58:49.101155 7276 leader_election.cc:334] T 00000000000000000000000000000000 P e84368bd28c54795852888b3d9163a75 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 0aeea5ec3a114f8c94fd2d2012fe312b (127.0.98.62:41421): Network error: Client connection negotiation failed: client connection to 127.0.98.62:41421: connect: Connection refused (error 111)
I20260926 03:58:49.101227 7809 consensus_queue.cc:237] T 00000000000000000000000000000000 P e84368bd28c54795852888b3d9163a75 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0aeea5ec3a114f8c94fd2d2012fe312b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41421 } } peers { permanent_uuid: "3f6ed0be592b4c6da0be1ae5feb24ff1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35627 } attrs { promote: false } } peers { permanent_uuid: "e84368bd28c54795852888b3d9163a75" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34077 } attrs { promote: false } }
I20260926 03:58:49.101570 7820 sys_catalog.cc:455] T 00000000000000000000000000000000 P e84368bd28c54795852888b3d9163a75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e84368bd28c54795852888b3d9163a75. Latest consensus state: current_term: 2 leader_uuid: "e84368bd28c54795852888b3d9163a75" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0aeea5ec3a114f8c94fd2d2012fe312b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41421 } } peers { permanent_uuid: "3f6ed0be592b4c6da0be1ae5feb24ff1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35627 } attrs { promote: false } } peers { permanent_uuid: "e84368bd28c54795852888b3d9163a75" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34077 } attrs { promote: false } } }
I20260926 03:58:49.101657 7820 sys_catalog.cc:458] T 00000000000000000000000000000000 P e84368bd28c54795852888b3d9163a75 [sys.catalog]: This master's current role is: LEADER
I20260926 03:58:49.101814 7822 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:58:49.102277 7822 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:58:49.102373 7822 catalog_manager.cc:1259] Loaded cluster ID: a989010329ac4d63b6f8b327d43ccef5
I20260926 03:58:49.102406 7822 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:58:49.102573 7822 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:58:49.102670 7822 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e84368bd28c54795852888b3d9163a75: Loaded TSK: 0
I20260926 03:58:49.102808 7822 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:58:49.145555 7290 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45454:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:58:49.147229 7290 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:58:49.148241 7112 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3f6ed0be592b4c6da0be1ae5feb24ff1 [term 2 FOLLOWER]: Refusing update from remote peer e84368bd28c54795852888b3d9163a75: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260926 03:58:49.148262 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:49.148141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41421 (local address 127.0.0.1:48094)
0926 03:58:49.148198 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:49.148226 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 03:58:49.148602 7809 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e84368bd28c54795852888b3d9163a75 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f6ed0be592b4c6da0be1ae5feb24ff1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35627 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260926 03:58:49.148763 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:49.148623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41421 (local address 127.0.0.1:48096)
0926 03:58:49.148708 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:49.148735 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260926 03:58:49.148844 7276 consensus_peers.cc:487] T 00000000000000000000000000000000 P e84368bd28c54795852888b3d9163a75 -> Peer 0aeea5ec3a114f8c94fd2d2012fe312b (127.0.98.62:41421): Couldn't send request to peer 0aeea5ec3a114f8c94fd2d2012fe312b. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41421: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 03:58:49.149246 7810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f6ed0be592b4c6da0be1ae5feb24ff1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e84368bd28c54795852888b3d9163a75. Latest consensus state: current_term: 2 leader_uuid: "e84368bd28c54795852888b3d9163a75" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0aeea5ec3a114f8c94fd2d2012fe312b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41421 } } peers { permanent_uuid: "3f6ed0be592b4c6da0be1ae5feb24ff1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35627 } attrs { promote: false } } peers { permanent_uuid: "e84368bd28c54795852888b3d9163a75" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34077 } attrs { promote: false } } }
I20260926 03:58:49.149345 7810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f6ed0be592b4c6da0be1ae5feb24ff1 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:58:49.149619 7823 mvcc.cc:205] Tried to move back new op lower bound from 7333458448989102080 to 7333458448799211520. Current Snapshot: MvccSnapshot[applied={T|T < 7333458448989102080}]
I20260926 03:58:49.149727 7809 sys_catalog.cc:455] T 00000000000000000000000000000000 P e84368bd28c54795852888b3d9163a75 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "e84368bd28c54795852888b3d9163a75" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0aeea5ec3a114f8c94fd2d2012fe312b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41421 } } peers { permanent_uuid: "3f6ed0be592b4c6da0be1ae5feb24ff1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35627 } attrs { promote: false } } peers { permanent_uuid: "e84368bd28c54795852888b3d9163a75" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34077 } attrs { promote: false } } }
I20260926 03:58:49.149816 7809 sys_catalog.cc:458] T 00000000000000000000000000000000 P e84368bd28c54795852888b3d9163a75 [sys.catalog]: This master's current role is: LEADER
I20260926 03:58:49.149907 7824 mvcc.cc:205] Tried to move back new op lower bound from 7333458448989102080 to 7333458448799211520. Current Snapshot: MvccSnapshot[applied={T|T < 7333458448989102080}]
I20260926 03:58:49.150019 7810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f6ed0be592b4c6da0be1ae5feb24ff1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "e84368bd28c54795852888b3d9163a75" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "0aeea5ec3a114f8c94fd2d2012fe312b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41421 } } peers { permanent_uuid: "3f6ed0be592b4c6da0be1ae5feb24ff1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 35627 } attrs { promote: false } } peers { permanent_uuid: "e84368bd28c54795852888b3d9163a75" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34077 } attrs { promote: false } } }
I20260926 03:58:49.150091 7810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f6ed0be592b4c6da0be1ae5feb24ff1 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 03:58:49.155222 7816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:49.151529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43115 (local address 127.0.0.1:60670)
0926 03:58:49.152002 (+ 473us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:49.152012 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:49.152055 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:49.152240 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:49.152241 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:49.152243 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:58:49.152290 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:49.152291 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:49.153079 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:49.153081 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:49.154218 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:49.154220 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:49.154984 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:49.154985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:49.155043 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:49.155048 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 03:58:49.155111 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 03:58:49.155153 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"mutex_wait_us":37}
I20260926 03:58:49.155248 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:49.151642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60670 (local address 127.0.98.2:43115)
0926 03:58:49.151840 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:49.151843 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:49.152044 (+ 201us) server_negotiation.cc:407] Connection header received
0926 03:58:49.152109 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:49.152110 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:49.152121 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:49.152354 (+ 233us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:58:49.152393 (+ 39us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:49.153001 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:49.154319 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:49.154895 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:49.154976 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:49.154988 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:49.155128 (+ 140us) server_negotiation.cc:299] Negotiation successful
0926 03:58:49.155175 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":63,"threads_started":1}
I20260926 03:58:49.155448 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:49.151480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37187 (local address 127.0.0.1:35760)
0926 03:58:49.152153 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:49.152165 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:49.152214 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:49.152445 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:49.152446 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:49.152448 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:58:49.152487 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:49.152488 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:49.153338 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:49.153339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:49.154391 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:49.154392 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:49.155212 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:49.155213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:49.155278 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:49.155282 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 03:58:49.155335 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 03:58:49.155381 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"mutex_wait_us":626}
I20260926 03:58:49.155469 7540 tablet_service.cc:1467] Processing CreateTablet for tablet 1a1053a9e836463480c7c378d9629337 (DEFAULT_TABLE table=test-table [id=a32ca90c369a42968e80f0c07a09304c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:58:49.155599 7540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a1053a9e836463480c7c378d9629337. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:58:49.156069 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:49.151480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43813 (local address 127.0.0.1:37052)
0926 03:58:49.151752 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:49.151763 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:49.151820 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:49.152210 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:49.152211 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:49.152213 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:58:49.152263 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:49.152264 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:49.153224 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:49.153224 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:49.155053 (+ 1829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:49.155055 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:49.155805 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:49.155805 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:49.155868 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:49.155871 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 03:58:49.155965 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 03:58:49.156009 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"mutex_wait_us":1028}
I20260926 03:58:49.156075 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:49.151652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35760 (local address 127.0.98.1:37187)
0926 03:58:49.151763 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:49.151766 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:49.152338 (+ 572us) server_negotiation.cc:407] Connection header received
0926 03:58:49.152368 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:49.152369 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:49.152381 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:49.152434 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:58:49.152539 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:49.153264 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:49.154479 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:49.155082 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:49.155168 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:49.155181 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:49.155918 (+ 737us) server_negotiation.cc:299] Negotiation successful
0926 03:58:49.156008 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":65,"threads_started":1}
I20260926 03:58:49.156086 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:49.151909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37052 (local address 127.0.98.3:43813)
0926 03:58:49.152036 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:49.152039 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:49.152049 (+ 10us) server_negotiation.cc:407] Connection header received
0926 03:58:49.152075 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:49.152076 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:49.152087 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:49.152202 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:58:49.152336 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:49.153132 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:49.155152 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:49.155713 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:49.155802 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:49.155816 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:49.155975 (+ 159us) server_negotiation.cc:299] Negotiation successful
0926 03:58:49.156019 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":60,"threads_started":1}
I20260926 03:58:49.156381 7676 tablet_service.cc:1467] Processing CreateTablet for tablet 1a1053a9e836463480c7c378d9629337 (DEFAULT_TABLE table=test-table [id=a32ca90c369a42968e80f0c07a09304c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:58:49.156488 7676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a1053a9e836463480c7c378d9629337. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:58:49.157160 7831 tablet_bootstrap.cc:492] T 1a1053a9e836463480c7c378d9629337 P 1e41eabd06d44a9ebab5ba2a76d5ea66: Bootstrap starting.
I20260926 03:58:49.157589 7831 tablet_bootstrap.cc:654] T 1a1053a9e836463480c7c378d9629337 P 1e41eabd06d44a9ebab5ba2a76d5ea66: Neither blocks nor log segments found. Creating new log.
I20260926 03:58:49.157935 7833 tablet_bootstrap.cc:492] T 1a1053a9e836463480c7c378d9629337 P 241844d2ef2b4666bd8513fdc36a4efb: Bootstrap starting.
I20260926 03:58:49.158207 7831 tablet_bootstrap.cc:492] T 1a1053a9e836463480c7c378d9629337 P 1e41eabd06d44a9ebab5ba2a76d5ea66: No bootstrap required, opened a new log
I20260926 03:58:49.158267 7831 ts_tablet_manager.cc:1397] T 1a1053a9e836463480c7c378d9629337 P 1e41eabd06d44a9ebab5ba2a76d5ea66: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 03:58:49.158257 7403 tablet_service.cc:1467] Processing CreateTablet for tablet 1a1053a9e836463480c7c378d9629337 (DEFAULT_TABLE table=test-table [id=a32ca90c369a42968e80f0c07a09304c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:58:49.158296 7833 tablet_bootstrap.cc:654] T 1a1053a9e836463480c7c378d9629337 P 241844d2ef2b4666bd8513fdc36a4efb: Neither blocks nor log segments found. Creating new log.
I20260926 03:58:49.158370 7403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a1053a9e836463480c7c378d9629337. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:58:49.158416 7831 raft_consensus.cc:348] T 1a1053a9e836463480c7c378d9629337 P 1e41eabd06d44a9ebab5ba2a76d5ea66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "226f08ac9eb34274ae439b88cea9050c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37187 } } peers { permanent_uuid: "241844d2ef2b4666bd8513fdc36a4efb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43813 } } peers { permanent_uuid: "1e41eabd06d44a9ebab5ba2a76d5ea66" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43115 } }
I20260926 03:58:49.158500 7831 raft_consensus.cc:374] T 1a1053a9e836463480c7c378d9629337 P 1e41eabd06d44a9ebab5ba2a76d5ea66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:58:49.158550 7831 raft_consensus.cc:729] T 1a1053a9e836463480c7c378d9629337 P 1e41eabd06d44a9ebab5ba2a76d5ea66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e41eabd06d44a9ebab5ba2a76d5ea66, State: Initialized, Role: FOLLOWER
I20260926 03:58:49.158599 7831 consensus_queue.cc:260] T 1a1053a9e836463480c7c378d9629337 P 1e41eabd06d44a9ebab5ba2a76d5ea66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "226f08ac9eb34274ae439b88cea9050c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37187 } } peers { permanent_uuid: "241844d2ef2b4666bd8513fdc36a4efb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43813 } } peers { permanent_uuid: "1e41eabd06d44a9ebab5ba2a76d5ea66" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43115 } }
I20260926 03:58:49.158730 7831 ts_tablet_manager.cc:1428] T 1a1053a9e836463480c7c378d9629337 P 1e41eabd06d44a9ebab5ba2a76d5ea66: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 03:58:49.158911 7833 tablet_bootstrap.cc:492] T 1a1053a9e836463480c7c378d9629337 P 241844d2ef2b4666bd8513fdc36a4efb: No bootstrap required, opened a new log
I20260926 03:58:49.159013 7833 ts_tablet_manager.cc:1397] T 1a1053a9e836463480c7c378d9629337 P 241844d2ef2b4666bd8513fdc36a4efb: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 03:58:49.159391 7833 raft_consensus.cc:348] T 1a1053a9e836463480c7c378d9629337 P 241844d2ef2b4666bd8513fdc36a4efb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "226f08ac9eb34274ae439b88cea9050c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37187 } } peers { permanent_uuid: "241844d2ef2b4666bd8513fdc36a4efb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43813 } } peers { permanent_uuid: "1e41eabd06d44a9ebab5ba2a76d5ea66" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43115 } }
I20260926 03:58:49.159520 7833 raft_consensus.cc:374] T 1a1053a9e836463480c7c378d9629337 P 241844d2ef2b4666bd8513fdc36a4efb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:58:49.159586 7833 raft_consensus.cc:729] T 1a1053a9e836463480c7c378d9629337 P 241844d2ef2b4666bd8513fdc36a4efb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 241844d2ef2b4666bd8513fdc36a4efb, State: Initialized, Role: FOLLOWER
I20260926 03:58:49.159600 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:49.158960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41421 (local address 127.0.98.2:57505)
0926 03:58:49.159527 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:49.159567 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":224,"threads_started":1}
I20260926 03:58:49.159823 7833 consensus_queue.cc:260] T 1a1053a9e836463480c7c378d9629337 P 241844d2ef2b4666bd8513fdc36a4efb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "226f08ac9eb34274ae439b88cea9050c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37187 } } peers { permanent_uuid: "241844d2ef2b4666bd8513fdc36a4efb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43813 } } peers { permanent_uuid: "1e41eabd06d44a9ebab5ba2a76d5ea66" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43115 } }
I20260926 03:58:49.160147 7833 ts_tablet_manager.cc:1428] T 1a1053a9e836463480c7c378d9629337 P 241844d2ef2b4666bd8513fdc36a4efb: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 03:58:49.160156 7837 tablet_bootstrap.cc:492] T 1a1053a9e836463480c7c378d9629337 P 226f08ac9eb34274ae439b88cea9050c: Bootstrap starting.
I20260926 03:58:49.160539 7837 tablet_bootstrap.cc:654] T 1a1053a9e836463480c7c378d9629337 P 226f08ac9eb34274ae439b88cea9050c: Neither blocks nor log segments found. Creating new log.
I20260926 03:58:49.160694 7742 heartbeater.cc:503] Master 127.0.98.60:34077 was elected leader, sending a full tablet report...
I20260926 03:58:49.160773 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:49.160394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41421 (local address 127.0.98.3:40565)
0926 03:58:49.160556 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:49.160729 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":75,"threads_started":1}
I20260926 03:58:49.161109 7837 tablet_bootstrap.cc:492] T 1a1053a9e836463480c7c378d9629337 P 226f08ac9eb34274ae439b88cea9050c: No bootstrap required, opened a new log
I20260926 03:58:49.161190 7837 ts_tablet_manager.cc:1397] T 1a1053a9e836463480c7c378d9629337 P 226f08ac9eb34274ae439b88cea9050c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 03:58:49.161249 7606 heartbeater.cc:503] Master 127.0.98.60:34077 was elected leader, sending a full tablet report...
I20260926 03:58:49.161310 7837 raft_consensus.cc:348] T 1a1053a9e836463480c7c378d9629337 P 226f08ac9eb34274ae439b88cea9050c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "226f08ac9eb34274ae439b88cea9050c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37187 } } peers { permanent_uuid: "241844d2ef2b4666bd8513fdc36a4efb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43813 } } peers { permanent_uuid: "1e41eabd06d44a9ebab5ba2a76d5ea66" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43115 } }
I20260926 03:58:49.161351 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:49.161042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41421 (local address 127.0.98.3:57731)
0926 03:58:49.161197 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:49.161274 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":78,"threads_started":1}
I20260926 03:58:49.161392 7837 raft_consensus.cc:374] T 1a1053a9e836463480c7c378d9629337 P 226f08ac9eb34274ae439b88cea9050c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:58:49.161414 7837 raft_consensus.cc:729] T 1a1053a9e836463480c7c378d9629337 P 226f08ac9eb34274ae439b88cea9050c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 226f08ac9eb34274ae439b88cea9050c, State: Initialized, Role: FOLLOWER
I20260926 03:58:49.161454 7837 consensus_queue.cc:260] T 1a1053a9e836463480c7c378d9629337 P 226f08ac9eb34274ae439b88cea9050c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "226f08ac9eb34274ae439b88cea9050c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37187 } } peers { permanent_uuid: "241844d2ef2b4666bd8513fdc36a4efb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43813 } } peers { permanent_uuid: "1e41eabd06d44a9ebab5ba2a76d5ea66" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43115 } }
I20260926 03:58:49.161621 7837 ts_tablet_manager.cc:1428] T 1a1053a9e836463480c7c378d9629337 P 226f08ac9eb34274ae439b88cea9050c: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 03:58:49.161804 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:49.161711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41421 (local address 127.0.98.2:46625)
0926 03:58:49.161749 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:49.161775 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260926 03:58:49.162120 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:49.161823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41421 (local address 127.0.98.1:55701)
0926 03:58:49.161982 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:49.162080 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":70,"threads_started":1}
I20260926 03:58:49.162457 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:49.162343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41421 (local address 127.0.98.1:51175)
0926 03:58:49.162412 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:49.162436 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260926 03:58:49.377920 7472 tablet.cc:2369] T 1a1053a9e836463480c7c378d9629337 P 226f08ac9eb34274ae439b88cea9050c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:58:49.407469 7745 tablet.cc:2369] T 1a1053a9e836463480c7c378d9629337 P 241844d2ef2b4666bd8513fdc36a4efb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:58:49.478628 7835 raft_consensus.cc:482] T 1a1053a9e836463480c7c378d9629337 P 1e41eabd06d44a9ebab5ba2a76d5ea66 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:58:49.478780 7835 raft_consensus.cc:504] T 1a1053a9e836463480c7c378d9629337 P 1e41eabd06d44a9ebab5ba2a76d5ea66 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "226f08ac9eb34274ae439b88cea9050c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37187 } } peers { permanent_uuid: "241844d2ef2b4666bd8513fdc36a4efb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43813 } } peers { permanent_uuid: "1e41eabd06d44a9ebab5ba2a76d5ea66" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43115 } }
I20260926 03:58:49.479012 7835 leader_election.cc:288] T 1a1053a9e836463480c7c378d9629337 P 1e41eabd06d44a9ebab5ba2a76d5ea66 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 226f08ac9eb34274ae439b88cea9050c (127.0.98.1:37187), 241844d2ef2b4666bd8513fdc36a4efb (127.0.98.3:43813)
I20260926 03:58:49.479233 7423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1053a9e836463480c7c378d9629337" candidate_uuid: "1e41eabd06d44a9ebab5ba2a76d5ea66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "226f08ac9eb34274ae439b88cea9050c" is_pre_election: true
I20260926 03:58:49.479285 7696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1053a9e836463480c7c378d9629337" candidate_uuid: "1e41eabd06d44a9ebab5ba2a76d5ea66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "241844d2ef2b4666bd8513fdc36a4efb" is_pre_election: true
I20260926 03:58:49.479364 7423 raft_consensus.cc:2454] T 1a1053a9e836463480c7c378d9629337 P 226f08ac9eb34274ae439b88cea9050c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e41eabd06d44a9ebab5ba2a76d5ea66 in term 0.
I20260926 03:58:49.479362 7696 raft_consensus.cc:2454] T 1a1053a9e836463480c7c378d9629337 P 241844d2ef2b4666bd8513fdc36a4efb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1e41eabd06d44a9ebab5ba2a76d5ea66 in term 0.
I20260926 03:58:49.479558 7495 leader_election.cc:302] T 1a1053a9e836463480c7c378d9629337 P 1e41eabd06d44a9ebab5ba2a76d5ea66 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e41eabd06d44a9ebab5ba2a76d5ea66, 241844d2ef2b4666bd8513fdc36a4efb; no voters:
I20260926 03:58:49.479701 7835 raft_consensus.cc:2781] T 1a1053a9e836463480c7c378d9629337 P 1e41eabd06d44a9ebab5ba2a76d5ea66 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:58:49.479758 7835 raft_consensus.cc:482] T 1a1053a9e836463480c7c378d9629337 P 1e41eabd06d44a9ebab5ba2a76d5ea66 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:58:49.479773 7835 raft_consensus.cc:3037] T 1a1053a9e836463480c7c378d9629337 P 1e41eabd06d44a9ebab5ba2a76d5ea66 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:58:49.480484 7835 raft_consensus.cc:504] T 1a1053a9e836463480c7c378d9629337 P 1e41eabd06d44a9ebab5ba2a76d5ea66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "226f08ac9eb34274ae439b88cea9050c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37187 } } peers { permanent_uuid: "241844d2ef2b4666bd8513fdc36a4efb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43813 } } peers { permanent_uuid: "1e41eabd06d44a9ebab5ba2a76d5ea66" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43115 } }
I20260926 03:58:49.480707 7835 leader_election.cc:288] T 1a1053a9e836463480c7c378d9629337 P 1e41eabd06d44a9ebab5ba2a76d5ea66 [CANDIDATE]: Term 1 election: Requested vote from peers 226f08ac9eb34274ae439b88cea9050c (127.0.98.1:37187), 241844d2ef2b4666bd8513fdc36a4efb (127.0.98.3:43813)
I20260926 03:58:49.480808 7696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1053a9e836463480c7c378d9629337" candidate_uuid: "1e41eabd06d44a9ebab5ba2a76d5ea66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "241844d2ef2b4666bd8513fdc36a4efb"
I20260926 03:58:49.480809 7423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1053a9e836463480c7c378d9629337" candidate_uuid: "1e41eabd06d44a9ebab5ba2a76d5ea66" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "226f08ac9eb34274ae439b88cea9050c"
I20260926 03:58:49.480862 7696 raft_consensus.cc:3037] T 1a1053a9e836463480c7c378d9629337 P 241844d2ef2b4666bd8513fdc36a4efb [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:58:49.480885 7423 raft_consensus.cc:3037] T 1a1053a9e836463480c7c378d9629337 P 226f08ac9eb34274ae439b88cea9050c [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:58:49.481407 7696 raft_consensus.cc:2454] T 1a1053a9e836463480c7c378d9629337 P 241844d2ef2b4666bd8513fdc36a4efb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e41eabd06d44a9ebab5ba2a76d5ea66 in term 1.
I20260926 03:58:49.481559 7423 raft_consensus.cc:2454] T 1a1053a9e836463480c7c378d9629337 P 226f08ac9eb34274ae439b88cea9050c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1e41eabd06d44a9ebab5ba2a76d5ea66 in term 1.
I20260926 03:58:49.481591 7495 leader_election.cc:302] T 1a1053a9e836463480c7c378d9629337 P 1e41eabd06d44a9ebab5ba2a76d5ea66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1e41eabd06d44a9ebab5ba2a76d5ea66, 241844d2ef2b4666bd8513fdc36a4efb; no voters:
I20260926 03:58:49.481680 7835 raft_consensus.cc:2781] T 1a1053a9e836463480c7c378d9629337 P 1e41eabd06d44a9ebab5ba2a76d5ea66 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:58:49.481838 7835 raft_consensus.cc:686] T 1a1053a9e836463480c7c378d9629337 P 1e41eabd06d44a9ebab5ba2a76d5ea66 [term 1 LEADER]: Becoming Leader. State: Replica: 1e41eabd06d44a9ebab5ba2a76d5ea66, State: Running, Role: LEADER
I20260926 03:58:49.481912 7835 consensus_queue.cc:237] T 1a1053a9e836463480c7c378d9629337 P 1e41eabd06d44a9ebab5ba2a76d5ea66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "226f08ac9eb34274ae439b88cea9050c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37187 } } peers { permanent_uuid: "241844d2ef2b4666bd8513fdc36a4efb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43813 } } peers { permanent_uuid: "1e41eabd06d44a9ebab5ba2a76d5ea66" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43115 } }
I20260926 03:58:49.482220 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:49.482094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41421 (local address 127.0.98.2:44699)
0926 03:58:49.482145 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:49.482186 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260926 03:58:49.482478 7290 catalog_manager.cc:5495] T 1a1053a9e836463480c7c378d9629337 P 1e41eabd06d44a9ebab5ba2a76d5ea66 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1e41eabd06d44a9ebab5ba2a76d5ea66 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "1e41eabd06d44a9ebab5ba2a76d5ea66" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "226f08ac9eb34274ae439b88cea9050c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "241844d2ef2b4666bd8513fdc36a4efb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1e41eabd06d44a9ebab5ba2a76d5ea66" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43115 } health_report { overall_health: HEALTHY } } }
I20260926 03:58:49.509441 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:49.507141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60686 (local address 127.0.98.2:43115)
0926 03:58:49.507212 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:49.507214 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:49.507225 (+ 11us) server_negotiation.cc:407] Connection header received
0926 03:58:49.507254 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:49.507255 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:49.507272 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:49.507343 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:58:49.507484 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:49.507964 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:49.508756 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:49.508948 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:49.509025 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:49.509170 (+ 145us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:58:49.509208 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:58:49.509270 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:49.509326 (+ 56us) server_negotiation.cc:299] Negotiation successful
0926 03:58:49.509350 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 03:58:49.510366 7696 raft_consensus.cc:1261] T 1a1053a9e836463480c7c378d9629337 P 241844d2ef2b4666bd8513fdc36a4efb [term 1 FOLLOWER]: Refusing update from remote peer 1e41eabd06d44a9ebab5ba2a76d5ea66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:58:49.510387 7423 raft_consensus.cc:1261] T 1a1053a9e836463480c7c378d9629337 P 226f08ac9eb34274ae439b88cea9050c [term 1 FOLLOWER]: Refusing update from remote peer 1e41eabd06d44a9ebab5ba2a76d5ea66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:58:49.510604 7844 consensus_queue.cc:1035] T 1a1053a9e836463480c7c378d9629337 P 1e41eabd06d44a9ebab5ba2a76d5ea66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "241844d2ef2b4666bd8513fdc36a4efb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:58:49.510717 7835 consensus_queue.cc:1035] T 1a1053a9e836463480c7c378d9629337 P 1e41eabd06d44a9ebab5ba2a76d5ea66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "226f08ac9eb34274ae439b88cea9050c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:58:49.511170 7469 heartbeater.cc:503] Master 127.0.98.60:34077 was elected leader, sending a full tablet report...
I20260926 03:58:49.511433 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:49.511324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41421 (local address 127.0.98.1:34839)
0926 03:58:49.511359 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:49.511399 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260926 03:58:49.512331 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:49.512157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41421 (local address 127.0.98.3:44607)
0926 03:58:49.512269 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:49.512298 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260926 03:58:49.512518 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:49.512290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41421 (local address 127.0.98.2:49985)
0926 03:58:49.512459 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:49.512490 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151}
I20260926 03:58:49.513275 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:49.513104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41421 (local address 127.0.98.1:52505)
0926 03:58:49.513222 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:49.513249 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260926 03:58:49.513883 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:49.513758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41421 (local address 127.0.98.2:46895)
0926 03:58:49.513822 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:49.513848 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260926 03:58:49.514195 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:49.514014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41421 (local address 127.0.98.3:36555)
0926 03:58:49.514134 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:49.514164 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41421: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260926 03:58:49.514467 7290 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45454:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:58:49.514562 7290 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45454:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:58:49.515816 7290 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e84368bd28c54795852888b3d9163a75: Sending DeleteTablet for 3 replicas of tablet 1a1053a9e836463480c7c378d9629337
I20260926 03:58:49.516139 7540 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a1053a9e836463480c7c378d9629337 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:58:49 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:60670
I20260926 03:58:49.516180 7403 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a1053a9e836463480c7c378d9629337 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:58:49 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:35760
I20260926 03:58:49.516305 7676 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a1053a9e836463480c7c378d9629337 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:58:49 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:37052
I20260926 03:58:49.516412 7858 tablet_replica.cc:330] T 1a1053a9e836463480c7c378d9629337 P 226f08ac9eb34274ae439b88cea9050c: stopping tablet replica
I20260926 03:58:49.516487 7858 raft_consensus.cc:2229] T 1a1053a9e836463480c7c378d9629337 P 226f08ac9eb34274ae439b88cea9050c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:58:49.516482 7857 tablet_replica.cc:330] T 1a1053a9e836463480c7c378d9629337 P 1e41eabd06d44a9ebab5ba2a76d5ea66: stopping tablet replica
I20260926 03:58:49.516539 7858 raft_consensus.cc:2258] T 1a1053a9e836463480c7c378d9629337 P 226f08ac9eb34274ae439b88cea9050c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:58:49.516541 7857 raft_consensus.cc:2229] T 1a1053a9e836463480c7c378d9629337 P 1e41eabd06d44a9ebab5ba2a76d5ea66 [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:58:49.516572 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 7252
I20260926 03:58:49.516597 7857 raft_consensus.cc:2258] T 1a1053a9e836463480c7c378d9629337 P 1e41eabd06d44a9ebab5ba2a76d5ea66 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:58:49.518311 7857 ts_tablet_manager.cc:1905] T 1a1053a9e836463480c7c378d9629337 P 1e41eabd06d44a9ebab5ba2a76d5ea66: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:58:49.520318 7857 ts_tablet_manager.cc:1918] T 1a1053a9e836463480c7c378d9629337 P 1e41eabd06d44a9ebab5ba2a76d5ea66: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:58:49.520388 7857 log.cc:1196] T 1a1053a9e836463480c7c378d9629337 P 1e41eabd06d44a9ebab5ba2a76d5ea66: Deleting WAL directory at /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-1/wal/wals/1a1053a9e836463480c7c378d9629337
I20260926 03:58:49.520663 7857 ts_tablet_manager.cc:1939] T 1a1053a9e836463480c7c378d9629337 P 1e41eabd06d44a9ebab5ba2a76d5ea66: Deleting consensus metadata
I20260926 03:58:49.520809 7859 tablet_replica.cc:330] T 1a1053a9e836463480c7c378d9629337 P 241844d2ef2b4666bd8513fdc36a4efb: stopping tablet replica
I20260926 03:58:49.520944 7859 raft_consensus.cc:2229] T 1a1053a9e836463480c7c378d9629337 P 241844d2ef2b4666bd8513fdc36a4efb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:58:49.520998 7859 raft_consensus.cc:2258] T 1a1053a9e836463480c7c378d9629337 P 241844d2ef2b4666bd8513fdc36a4efb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:58:49.521144 7274 catalog_manager.cc:4841] TS 1e41eabd06d44a9ebab5ba2a76d5ea66 (127.0.98.2:43115): tablet 1a1053a9e836463480c7c378d9629337 (table test-table [id=a32ca90c369a42968e80f0c07a09304c]) successfully deleted
I20260926 03:58:49.521306 7859 ts_tablet_manager.cc:1905] T 1a1053a9e836463480c7c378d9629337 P 241844d2ef2b4666bd8513fdc36a4efb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:58:49.522331 7811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:49.522197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37187 (local address 127.0.0.1:35770)
0926 03:58:49.522260 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:49.522299 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260926 03:58:49.522379 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 7477
W20260926 03:58:49.523222 7275 catalog_manager.cc:4584] TS 226f08ac9eb34274ae439b88cea9050c (127.0.98.1:37187): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1a1053a9e836463480c7c378d9629337: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37187: connect: Connection refused (error 111)
I20260926 03:58:49.528090 7859 ts_tablet_manager.cc:1918] T 1a1053a9e836463480c7c378d9629337 P 241844d2ef2b4666bd8513fdc36a4efb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:58:49.528167 7859 log.cc:1196] T 1a1053a9e836463480c7c378d9629337 P 241844d2ef2b4666bd8513fdc36a4efb: Deleting WAL directory at /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790395095474643-392-0/minicluster-data/ts-2/wal/wals/1a1053a9e836463480c7c378d9629337
I20260926 03:58:49.528406 7859 ts_tablet_manager.cc:1939] T 1a1053a9e836463480c7c378d9629337 P 241844d2ef2b4666bd8513fdc36a4efb: Deleting consensus metadata
I20260926 03:58:49.529088 7276 catalog_manager.cc:4841] TS 241844d2ef2b4666bd8513fdc36a4efb (127.0.98.3:43813): tablet 1a1053a9e836463480c7c378d9629337 (table test-table [id=a32ca90c369a42968e80f0c07a09304c]) successfully deleted
I20260926 03:58:49.529524 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 7613
I20260926 03:58:49.535218 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 6962
I20260926 03:58:49.540977 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 7156
2026-09-26T03:58:49Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (4900 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:58:49 dist-test-slave-nhpw krb5kdc[7862](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:58:49 dist-test-slave-nhpw krb5kdc[7862](info): set up 2 sockets
Sep 26 03:58:49 dist-test-slave-nhpw krb5kdc[7862](info): commencing operation
krb5kdc: starting...
W20260926 03:58:50.585094 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/krb5kdc/test-user.keytab.
Sep 26 03:58:50 dist-test-slave-nhpw krb5kdc[7862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:58:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:58:50Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260926 03:58:50.703846 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35257
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35257
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:58:50.781924 7875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:58:50.782174 7875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:58:50.782208 7875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:58:50.783393 7875 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:58:50.783450 7875 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 03:58:50.783473 7875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:58:50.783502 7875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:58:50.783558 7875 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:58:50.783586 7875 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:58:50.785003 7875 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35257
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:58:50.785269 7875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:58:50.785476 7875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:58:50.790148 7883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:50.790174 7881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:50.790174 7875 server_base.cc:1034] running on GCE node
W20260926 03:58:50.790407 7880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:50.790649 7875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:58:50.790884 7875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:58:50.792021 7875 hybrid_clock.cc:648] HybridClock initialized: now 1790395130791996 us; error 41 us; skew 500 ppm
Sep 26 03:58:50 dist-test-slave-nhpw krb5kdc[7862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395130, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:58:50.798242 7875 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260926 03:58:50.798672 7875 webserver.cc:460] Webserver started at http://127.0.98.62:41255/ using document root <none> and password file <none>
I20260926 03:58:50.798851 7875 fs_manager.cc:359] Metadata directory not provided
I20260926 03:58:50.798889 7875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:58:50.798971 7875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:58:50.799912 7875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/minicluster-data/master-0/data/instance:
uuid: "bac06df8dabe430ca2575d7abbe118ac"
format_stamp: "Formatted at 2026-09-26 03:58:50 on dist-test-slave-nhpw"
server_key: "921105a93fad34c058af40d5fb185b7d"
server_key_iv: "ec107261d7e1f78b919b1d73ec36752e"
server_key_version: "encryptionkey@0"
I20260926 03:58:50.800199 7875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/minicluster-data/master-0/wal/instance:
uuid: "bac06df8dabe430ca2575d7abbe118ac"
format_stamp: "Formatted at 2026-09-26 03:58:50 on dist-test-slave-nhpw"
server_key: "921105a93fad34c058af40d5fb185b7d"
server_key_iv: "ec107261d7e1f78b919b1d73ec36752e"
server_key_version: "encryptionkey@0"
I20260926 03:58:50.801468 7875 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 03:58:50.802086 7890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:50.802287 7875 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 03:58:50.802366 7875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/minicluster-data/master-0/wal
uuid: "bac06df8dabe430ca2575d7abbe118ac"
format_stamp: "Formatted at 2026-09-26 03:58:50 on dist-test-slave-nhpw"
server_key: "921105a93fad34c058af40d5fb185b7d"
server_key_iv: "ec107261d7e1f78b919b1d73ec36752e"
server_key_version: "encryptionkey@0"
I20260926 03:58:50.802415 7875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:58:50.811209 7875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:58:50.813083 7875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:58:50.813184 7875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:58:50.817402 7875 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35257
I20260926 03:58:50.817474 7952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35257 every 8 connection(s)
I20260926 03:58:50.817763 7875 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
I20260926 03:58:50.818120 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 7875
I20260926 03:58:50.818230 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/minicluster-data/master-0/wal/instance
I20260926 03:58:50.818408 392 external_mini_cluster.cc:1428] Setting key b83b2f8315871eea72856affd1327157
I20260926 03:58:50.818500 7953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:58:50.821297 7953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bac06df8dabe430ca2575d7abbe118ac: Bootstrap starting.
I20260926 03:58:50.821921 7953 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bac06df8dabe430ca2575d7abbe118ac: Neither blocks nor log segments found. Creating new log.
I20260926 03:58:50.822194 7953 log.cc:824] T 00000000000000000000000000000000 P bac06df8dabe430ca2575d7abbe118ac: Log is configured to *not* fsync() on all Append() calls
I20260926 03:58:50.822849 7953 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bac06df8dabe430ca2575d7abbe118ac: No bootstrap required, opened a new log
Sep 26 03:58:50 dist-test-slave-nhpw krb5kdc[7862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395130, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 03:58:50.825037 7953 raft_consensus.cc:348] T 00000000000000000000000000000000 P bac06df8dabe430ca2575d7abbe118ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bac06df8dabe430ca2575d7abbe118ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35257 } }
I20260926 03:58:50.825165 7953 raft_consensus.cc:374] T 00000000000000000000000000000000 P bac06df8dabe430ca2575d7abbe118ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:58:50.825188 7953 raft_consensus.cc:729] T 00000000000000000000000000000000 P bac06df8dabe430ca2575d7abbe118ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bac06df8dabe430ca2575d7abbe118ac, State: Initialized, Role: FOLLOWER
I20260926 03:58:50.825335 7953 consensus_queue.cc:260] T 00000000000000000000000000000000 P bac06df8dabe430ca2575d7abbe118ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bac06df8dabe430ca2575d7abbe118ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35257 } }
I20260926 03:58:50.825394 7953 raft_consensus.cc:388] T 00000000000000000000000000000000 P bac06df8dabe430ca2575d7abbe118ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:58:50.825451 7953 raft_consensus.cc:482] T 00000000000000000000000000000000 P bac06df8dabe430ca2575d7abbe118ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:58:50.825505 7953 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bac06df8dabe430ca2575d7abbe118ac [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:58:50.826103 7953 raft_consensus.cc:504] T 00000000000000000000000000000000 P bac06df8dabe430ca2575d7abbe118ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bac06df8dabe430ca2575d7abbe118ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35257 } }
I20260926 03:58:50.826212 7953 leader_election.cc:302] T 00000000000000000000000000000000 P bac06df8dabe430ca2575d7abbe118ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bac06df8dabe430ca2575d7abbe118ac; no voters:
I20260926 03:58:50.826368 7953 leader_election.cc:288] T 00000000000000000000000000000000 P bac06df8dabe430ca2575d7abbe118ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:58:50.826401 7958 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bac06df8dabe430ca2575d7abbe118ac [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:58:50.826512 7958 raft_consensus.cc:686] T 00000000000000000000000000000000 P bac06df8dabe430ca2575d7abbe118ac [term 1 LEADER]: Becoming Leader. State: Replica: bac06df8dabe430ca2575d7abbe118ac, State: Running, Role: LEADER
I20260926 03:58:50.826592 7958 consensus_queue.cc:237] T 00000000000000000000000000000000 P bac06df8dabe430ca2575d7abbe118ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bac06df8dabe430ca2575d7abbe118ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35257 } }
I20260926 03:58:50.826632 7953 sys_catalog.cc:564] T 00000000000000000000000000000000 P bac06df8dabe430ca2575d7abbe118ac [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:58:50.826902 7958 sys_catalog.cc:455] T 00000000000000000000000000000000 P bac06df8dabe430ca2575d7abbe118ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader bac06df8dabe430ca2575d7abbe118ac. Latest consensus state: current_term: 1 leader_uuid: "bac06df8dabe430ca2575d7abbe118ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bac06df8dabe430ca2575d7abbe118ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35257 } } }
I20260926 03:58:50.826985 7958 sys_catalog.cc:458] T 00000000000000000000000000000000 P bac06df8dabe430ca2575d7abbe118ac [sys.catalog]: This master's current role is: LEADER
I20260926 03:58:50.827193 7964 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:58:50.826910 7960 sys_catalog.cc:455] T 00000000000000000000000000000000 P bac06df8dabe430ca2575d7abbe118ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bac06df8dabe430ca2575d7abbe118ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bac06df8dabe430ca2575d7abbe118ac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35257 } } }
I20260926 03:58:50.827397 7960 sys_catalog.cc:458] T 00000000000000000000000000000000 P bac06df8dabe430ca2575d7abbe118ac [sys.catalog]: This master's current role is: LEADER
I20260926 03:58:50.827708 7964 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:58:50.828867 7956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:50.818884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33464 (local address 127.0.98.62:35257)
0926 03:58:50.819066 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:50.819074 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:50.819108 (+ 34us) server_negotiation.cc:407] Connection header received
0926 03:58:50.819512 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:50.819518 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:50.819561 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:50.819702 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:50.820415 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:50.821316 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:50.822095 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:50.822512 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:50.822640 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:50.825127 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:50.825133 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:50.825134 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:50.825148 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:50.827184 (+ 2036us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:50.827583 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:50.827586 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:50.827586 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:50.827699 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:50.827874 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:50.827875 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:50.827875 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:50.828173 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:50.828316 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:50.828436 (+ 120us) server_negotiation.cc:299] Negotiation successful
0926 03:58:50.828696 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":89,"threads_started":1}
I20260926 03:58:50.829406 7964 catalog_manager.cc:1347] Generated new cluster ID: f1bae4fdf25041f0a9e3ce8d1b9c090a
I20260926 03:58:50.829458 7964 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:58:50.841471 7964 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:58:50.842141 7964 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:58:50.849260 7964 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bac06df8dabe430ca2575d7abbe118ac: Generated new TSK 0
I20260926 03:58:50.849463 7964 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260926 03:58:50.879877 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35257
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:58:50.968118 7979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:58:50.968322 7979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:58:50.968348 7979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:58:50.969561 7979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:58:50.969609 7979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:58:50.969630 7979 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 03:58:50.969686 7979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 03:58:50.971148 7979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:58:50.971380 7979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:58:50.971621 7979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:58:50.977030 7984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:50.977005 7985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:50.977012 7987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:50.977169 7979 server_base.cc:1034] running on GCE node
I20260926 03:58:50.977363 7979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:58:50.977612 7979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:58:50.978787 7979 hybrid_clock.cc:648] HybridClock initialized: now 1790395130978767 us; error 32 us; skew 500 ppm
Sep 26 03:58:50 dist-test-slave-nhpw krb5kdc[7862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395130, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:58:50.981388 7979 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260926 03:58:50.981819 7979 webserver.cc:460] Webserver started at http://127.0.98.1:36495/ using document root <none> and password file <none>
I20260926 03:58:50.982044 7979 fs_manager.cc:359] Metadata directory not provided
I20260926 03:58:50.982089 7979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:58:50.982234 7979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:58:50.983608 7979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/minicluster-data/ts-0/data/instance:
uuid: "cc1f89d3f0cd49588788997ecba18080"
format_stamp: "Formatted at 2026-09-26 03:58:50 on dist-test-slave-nhpw"
server_key: "689b213d02361286953b743084bd0baa"
server_key_iv: "54df63be9ef1436408dbd743610841b8"
server_key_version: "encryptionkey@0"
I20260926 03:58:50.984046 7979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/minicluster-data/ts-0/wal/instance:
uuid: "cc1f89d3f0cd49588788997ecba18080"
format_stamp: "Formatted at 2026-09-26 03:58:50 on dist-test-slave-nhpw"
server_key: "689b213d02361286953b743084bd0baa"
server_key_iv: "54df63be9ef1436408dbd743610841b8"
server_key_version: "encryptionkey@0"
I20260926 03:58:50.985877 7979 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260926 03:58:50.986711 7994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:50.986868 7979 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 03:58:50.986968 7979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/minicluster-data/ts-0/wal
uuid: "cc1f89d3f0cd49588788997ecba18080"
format_stamp: "Formatted at 2026-09-26 03:58:50 on dist-test-slave-nhpw"
server_key: "689b213d02361286953b743084bd0baa"
server_key_iv: "54df63be9ef1436408dbd743610841b8"
server_key_version: "encryptionkey@0"
I20260926 03:58:50.987056 7979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:58:50.995635 7979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:58:50.997455 7979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:58:50.997614 7979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:58:50.998081 7979 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:58:50.998145 7979 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:50.998190 7979 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:58:50.998224 7979 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 03:58:51 dist-test-slave-nhpw krb5kdc[7862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395130, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 03:58:51.005482 7979 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43027
I20260926 03:58:51.005555 8109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43027 every 8 connection(s)
I20260926 03:58:51.005913 7979 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
I20260926 03:58:51.008517 7956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:50.998614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42005 (local address 127.0.98.62:35257)
0926 03:58:50.998688 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:50.998690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:50.998959 (+ 269us) server_negotiation.cc:407] Connection header received
0926 03:58:50.999676 (+ 717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:50.999677 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:50.999687 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:50.999731 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:51.000612 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:51.001440 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:51.003108 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:51.003451 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:51.003525 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:51.005841 (+ 2316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:51.005843 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:51.005843 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:51.005852 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:51.007371 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:51.007727 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:51.007728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:51.007728 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:51.007779 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:51.007977 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:51.007977 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:51.007978 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:51.008074 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:51.008145 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:51.008385 (+ 240us) server_negotiation.cc:299] Negotiation successful
0926 03:58:51.008453 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 03:58:51.008646 8003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:50.998590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35257 (local address 127.0.98.1:42005)
0926 03:58:50.998790 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:50.998836 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:50.999564 (+ 728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:50.999808 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:50.999812 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:51.000218 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:51.000504 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:51.000511 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:51.001541 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:51.001542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:51.003006 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:51.003009 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:51.003530 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:51.003532 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:51.003655 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:51.004164 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:51.004172 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:51.005744 (+ 1572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:58:51.007442 (+ 1698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:51.007447 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:51.007448 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:51.007660 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:51.007834 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:51.007835 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:51.007835 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:51.007924 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:51.008159 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:58:51.008163 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:58:51.008274 (+ 111us) client_negotiation.cc:769] Sending connection context
0926 03:58:51.008358 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 03:58:51.008470 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":103,"threads_started":1}
I20260926 03:58:51.009143 8110 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35257
I20260926 03:58:51.009227 8110 heartbeater.cc:464] Registering TS with master...
I20260926 03:58:51.009397 8110 heartbeater.cc:511] Master 127.0.98.62:35257 requested a full tablet report, sending...
I20260926 03:58:51.009958 7907 ts_manager.cc:194] Registered new tserver with Master: cc1f89d3f0cd49588788997ecba18080 (127.0.98.1:43027)
I20260926 03:58:51.010599 7907 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42005
I20260926 03:58:51.014976 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 7979
I20260926 03:58:51.015138 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/minicluster-data/ts-0/wal/instance
I20260926 03:58:51.015349 392 external_mini_cluster.cc:1428] Setting key 42b10b17281c38acbf115e1aae972180
I20260926 03:58:51.015380 7956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:51.010153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42705 (local address 127.0.98.62:35257)
0926 03:58:51.010195 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:51.010197 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:51.010312 (+ 115us) server_negotiation.cc:407] Connection header received
0926 03:58:51.010382 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:51.010383 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:51.010393 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:51.010436 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:51.010833 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:51.011345 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:51.012060 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:51.012266 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:51.012340 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:51.013194 (+ 854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:51.013196 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:51.013196 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:51.013203 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:51.014290 (+ 1087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:51.014637 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:51.014638 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:51.014639 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:51.014690 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:51.014905 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:51.014905 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:51.014906 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:51.015009 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:51.015085 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:51.015220 (+ 135us) server_negotiation.cc:299] Negotiation successful
0926 03:58:51.015293 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 03:58:51.015394 8116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:51.010108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35257 (local address 127.0.98.1:42705)
0926 03:58:51.010238 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:51.010247 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:51.010326 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:51.010449 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:51.010450 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:51.010618 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:51.010666 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:51.010667 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:51.011442 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:51.011443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:51.011965 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:51.011967 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:51.012338 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:51.012339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:51.012403 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:51.012838 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:51.012842 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:51.013094 (+ 252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:58:51.014375 (+ 1281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:51.014376 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:51.014376 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:51.014540 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:51.014766 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:51.014767 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:51.014767 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:51.014823 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:51.015077 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:58:51.015078 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:58:51.015120 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 03:58:51.015194 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 03:58:51.015292 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":68,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260926 03:58:51.039705 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35257
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:58:51.118440 8119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:58:51.118672 8119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:58:51.118710 8119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:58:51.119849 8119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:58:51.119913 8119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:58:51.119946 8119 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 03:58:51.119983 8119 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 03:58:51.121594 8119 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8119
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:58:51.121855 8119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:58:51.122080 8119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:58:51.127242 8127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:51.127278 8124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:51.127267 8125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:51.127425 8119 server_base.cc:1034] running on GCE node
I20260926 03:58:51.127642 8119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:58:51.127895 8119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:58:51.129034 8119 hybrid_clock.cc:648] HybridClock initialized: now 1790395131129025 us; error 28 us; skew 500 ppm
Sep 26 03:58:51 dist-test-slave-nhpw krb5kdc[7862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395131, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:58:51.131484 8119 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260926 03:58:51.131913 8119 webserver.cc:460] Webserver started at http://127.0.98.2:37747/ using document root <none> and password file <none>
I20260926 03:58:51.132108 8119 fs_manager.cc:359] Metadata directory not provided
I20260926 03:58:51.132148 8119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:58:51.132232 8119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:58:51.133244 8119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/minicluster-data/ts-1/data/instance:
uuid: "bcf2b7fca8ce4133ac3c49f74265078a"
format_stamp: "Formatted at 2026-09-26 03:58:51 on dist-test-slave-nhpw"
server_key: "ad6f13f63c18a14c3f7831d653925787"
server_key_iv: "90212197c7720b2c7b8ebc8d891b055f"
server_key_version: "encryptionkey@0"
I20260926 03:58:51.133543 8119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/minicluster-data/ts-1/wal/instance:
uuid: "bcf2b7fca8ce4133ac3c49f74265078a"
format_stamp: "Formatted at 2026-09-26 03:58:51 on dist-test-slave-nhpw"
server_key: "ad6f13f63c18a14c3f7831d653925787"
server_key_iv: "90212197c7720b2c7b8ebc8d891b055f"
server_key_version: "encryptionkey@0"
I20260926 03:58:51.134796 8119 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 03:58:51.135471 8134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:51.135650 8119 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 03:58:51.135735 8119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/minicluster-data/ts-1/wal
uuid: "bcf2b7fca8ce4133ac3c49f74265078a"
format_stamp: "Formatted at 2026-09-26 03:58:51 on dist-test-slave-nhpw"
server_key: "ad6f13f63c18a14c3f7831d653925787"
server_key_iv: "90212197c7720b2c7b8ebc8d891b055f"
server_key_version: "encryptionkey@0"
I20260926 03:58:51.135802 8119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:58:51.156176 8119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:58:51.158456 8119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:58:51.158608 8119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:58:51.159204 8119 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:58:51.159250 8119 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:51.159284 8119 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:58:51.159302 8119 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 03:58:51 dist-test-slave-nhpw krb5kdc[7862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395131, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 03:58:51.166316 8119 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34219
I20260926 03:58:51.166463 8249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34219 every 8 connection(s)
I20260926 03:58:51.166731 8119 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
I20260926 03:58:51.169112 7956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:51.159648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50421 (local address 127.0.98.62:35257)
0926 03:58:51.159715 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:51.159717 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:51.159896 (+ 179us) server_negotiation.cc:407] Connection header received
0926 03:58:51.160382 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:51.160383 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:51.160398 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:51.160481 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:51.161161 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:51.162030 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:51.163152 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:51.163414 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:51.163500 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:51.166485 (+ 2985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:51.166487 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:51.166487 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:51.166495 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:51.167921 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:51.168305 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:51.168306 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:51.168306 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:51.168353 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:51.168554 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:51.168554 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:51.168554 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:51.168698 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:51.168762 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:51.168964 (+ 202us) server_negotiation.cc:299] Negotiation successful
0926 03:58:51.169049 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 03:58:51.169333 8143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:51.159615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35257 (local address 127.0.98.2:50421)
0926 03:58:51.159769 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:51.159806 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:51.160292 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:51.160586 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:51.160592 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:51.160877 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:51.161084 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:51.161088 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:51.162129 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:51.162130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:51.163057 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:51.163060 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:51.163520 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:51.163521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:51.163649 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:51.164148 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:51.164160 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:51.166391 (+ 2231us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:58:51.167989 (+ 1598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:51.167994 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:51.167995 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:51.168240 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:51.168410 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:51.168411 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:51.168411 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:51.168497 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:51.168755 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:58:51.168758 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:58:51.168866 (+ 108us) client_negotiation.cc:769] Sending connection context
0926 03:58:51.168950 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 03:58:51.169093 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":81,"threads_started":1}
I20260926 03:58:51.169790 8250 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35257
I20260926 03:58:51.169869 8250 heartbeater.cc:464] Registering TS with master...
I20260926 03:58:51.170050 8250 heartbeater.cc:511] Master 127.0.98.62:35257 requested a full tablet report, sending...
I20260926 03:58:51.170452 7906 ts_manager.cc:194] Registered new tserver with Master: bcf2b7fca8ce4133ac3c49f74265078a (127.0.98.2:34219)
I20260926 03:58:51.170922 7906 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50421
I20260926 03:58:51.174433 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 8119
I20260926 03:58:51.174572 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/minicluster-data/ts-1/wal/instance
I20260926 03:58:51.174780 392 external_mini_cluster.cc:1428] Setting key 874539dc16328b6615521bfc79b87dad
I20260926 03:58:51.176492 7956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:51.170780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34051 (local address 127.0.98.62:35257)
0926 03:58:51.170837 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:51.170839 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:51.171017 (+ 178us) server_negotiation.cc:407] Connection header received
0926 03:58:51.171097 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:51.171098 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:51.171109 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:51.171162 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:51.171450 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:51.171908 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:51.172664 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:51.172863 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:51.172939 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:51.173812 (+ 873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:51.173814 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:51.173814 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:51.173820 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:51.175278 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:51.175772 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:51.175773 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:51.175773 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:51.175820 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:51.176043 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:51.176044 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:51.176044 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:51.176171 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:51.176222 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:51.176377 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 03:58:51.176434 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 03:58:51.176662 8256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:51.170748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35257 (local address 127.0.98.2:34051)
0926 03:58:51.170935 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:51.170944 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:51.171029 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:51.171167 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:51.171168 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:51.171320 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:51.171370 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:51.171371 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:51.171997 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:51.172031 (+ 34us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:51.172544 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:51.172546 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:51.172939 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:51.172940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:51.173023 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:51.173439 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:51.173444 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:51.173708 (+ 264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:58:51.175432 (+ 1724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:51.175433 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:51.175434 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:51.175698 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:51.175894 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:51.175895 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:51.175895 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:51.175965 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:51.176235 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:58:51.176236 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:58:51.176283 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 03:58:51.176361 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 03:58:51.176472 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":140,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260926 03:58:51.200088 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35257
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:58:51.281422 8259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:58:51.281627 8259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:58:51.281657 8259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:58:51.282843 8259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:58:51.282886 8259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:58:51.282907 8259 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 03:58:51.282924 8259 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 03:58:51.284479 8259 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35257
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:58:51.284751 8259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:58:51.284996 8259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:58:51.289808 8265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:51.289819 8267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:51.289798 8264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:51.290261 8259 server_base.cc:1034] running on GCE node
I20260926 03:58:51.290455 8259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:58:51.290700 8259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:58:51.291869 8259 hybrid_clock.cc:648] HybridClock initialized: now 1790395131291850 us; error 31 us; skew 500 ppm
Sep 26 03:58:51 dist-test-slave-nhpw krb5kdc[7862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395131, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:58:51.294440 8259 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260926 03:58:51.294886 8259 webserver.cc:460] Webserver started at http://127.0.98.3:39299/ using document root <none> and password file <none>
I20260926 03:58:51.295117 8259 fs_manager.cc:359] Metadata directory not provided
I20260926 03:58:51.295186 8259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:58:51.295336 8259 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:58:51.296373 8259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/minicluster-data/ts-2/data/instance:
uuid: "37dfcdbd7b6f4745a9c25c117f83aa74"
format_stamp: "Formatted at 2026-09-26 03:58:51 on dist-test-slave-nhpw"
server_key: "fa69201fc5b4e768c7ed915bf800bd1d"
server_key_iv: "d41ef182168e048721b77f7348601895"
server_key_version: "encryptionkey@0"
I20260926 03:58:51.296777 8259 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/minicluster-data/ts-2/wal/instance:
uuid: "37dfcdbd7b6f4745a9c25c117f83aa74"
format_stamp: "Formatted at 2026-09-26 03:58:51 on dist-test-slave-nhpw"
server_key: "fa69201fc5b4e768c7ed915bf800bd1d"
server_key_iv: "d41ef182168e048721b77f7348601895"
server_key_version: "encryptionkey@0"
I20260926 03:58:51.298143 8259 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 03:58:51.298975 8274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:51.299161 8259 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 03:58:51.299266 8259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/minicluster-data/ts-2/wal
uuid: "37dfcdbd7b6f4745a9c25c117f83aa74"
format_stamp: "Formatted at 2026-09-26 03:58:51 on dist-test-slave-nhpw"
server_key: "fa69201fc5b4e768c7ed915bf800bd1d"
server_key_iv: "d41ef182168e048721b77f7348601895"
server_key_version: "encryptionkey@0"
I20260926 03:58:51.299353 8259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:58:51.313560 8259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:58:51.315572 8259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:58:51.315728 8259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:58:51.316214 8259 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:58:51.316277 8259 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:51.316323 8259 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:58:51.316356 8259 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 03:58:51 dist-test-slave-nhpw krb5kdc[7862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395131, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 03:58:51.323491 8259 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35815
I20260926 03:58:51.323896 8259 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
I20260926 03:58:51.323868 8389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35815 every 8 connection(s)
I20260926 03:58:51.324834 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 8259
I20260926 03:58:51.324945 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/minicluster-data/ts-2/wal/instance
I20260926 03:58:51.325192 392 external_mini_cluster.cc:1428] Setting key d0430a35ef9ecd42edc7bb71d22a9737
I20260926 03:58:51.326252 7956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:51.316772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39465 (local address 127.0.98.62:35257)
0926 03:58:51.316819 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:51.316821 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:51.317218 (+ 397us) server_negotiation.cc:407] Connection header received
0926 03:58:51.317731 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:51.317732 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:51.317744 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:51.317802 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:51.318392 (+ 590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:51.318905 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:51.319907 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:51.320111 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:51.320197 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:51.323020 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:51.323026 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:51.323027 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:51.323034 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:51.324611 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:51.325144 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:51.325146 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:51.325146 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:51.325210 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:51.325514 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:51.325515 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:51.325515 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:51.325673 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:51.325758 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:51.326105 (+ 347us) server_negotiation.cc:299] Negotiation successful
0926 03:58:51.326182 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 03:58:51.326447 8283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:51.316791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35257 (local address 127.0.98.3:39465)
0926 03:58:51.316969 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:51.317030 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:51.317623 (+ 593us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:51.317870 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:51.317875 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:51.318119 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:51.318291 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:51.318295 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:51.319010 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:51.319011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:51.319813 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:51.319815 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:51.320208 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:51.320208 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:51.320363 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:51.320932 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:51.320942 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:51.322904 (+ 1962us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:58:51.324745 (+ 1841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:51.324752 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:51.324754 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:51.325046 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:51.325284 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:51.325285 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:51.325286 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:51.325427 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:51.325754 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:58:51.325759 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:58:51.325919 (+ 160us) client_negotiation.cc:769] Sending connection context
0926 03:58:51.326062 (+ 143us) client_negotiation.cc:240] Negotiation successful
0926 03:58:51.326224 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":83,"threads_started":1}
I20260926 03:58:51.326920 8390 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35257
I20260926 03:58:51.327008 8390 heartbeater.cc:464] Registering TS with master...
I20260926 03:58:51.327183 8390 heartbeater.cc:511] Master 127.0.98.62:35257 requested a full tablet report, sending...
I20260926 03:58:51.327548 7906 ts_manager.cc:194] Registered new tserver with Master: 37dfcdbd7b6f4745a9c25c117f83aa74 (127.0.98.3:35815)
I20260926 03:58:51.328086 7906 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39465
I20260926 03:58:51.333271 7956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:51.328001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52053 (local address 127.0.98.62:35257)
0926 03:58:51.328038 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:51.328040 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:51.328339 (+ 299us) server_negotiation.cc:407] Connection header received
0926 03:58:51.328384 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:51.328385 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:51.328395 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:51.328456 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:51.328799 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:51.329377 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:51.330068 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:51.330241 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:51.330296 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:51.331216 (+ 920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:51.331218 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:51.331218 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:51.331225 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:51.332355 (+ 1130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:51.332685 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:51.332686 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:51.332687 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:51.332740 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:51.332909 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:51.332910 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:51.332910 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:51.332994 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:51.333039 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:51.333163 (+ 124us) server_negotiation.cc:299] Negotiation successful
0926 03:58:51.333216 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 03:58:51.333284 8396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:51.327935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35257 (local address 127.0.98.3:52053)
0926 03:58:51.328271 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:51.328282 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:51.328333 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:51.328459 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:51.328460 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:51.328666 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:51.328721 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:51.328723 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:51.329485 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:51.329486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:51.329992 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:51.329994 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:51.330309 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:51.330310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:51.330355 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:51.330815 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:51.330819 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:51.331120 (+ 301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:58:51.332420 (+ 1300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:51.332421 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:51.332422 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:51.332574 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:51.332808 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:51.332809 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:51.332809 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:51.332860 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:51.333043 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:58:51.333044 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:58:51.333096 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 03:58:51.333147 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 03:58:51.333218 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":81,"threads_started":1}
I20260926 03:58:51.336913 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 03:58:51 dist-test-slave-nhpw krb5kdc[7862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395131, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 03:58:51 dist-test-slave-nhpw krb5kdc[7862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:58:51 dist-test-slave-nhpw krb5kdc[7862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 03:58:51.357484 7956 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:58:51.352041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33472 (local address 127.0.98.62:35257)
0926 03:58:51.352119 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:51.352121 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:51.352129 (+ 8us) server_negotiation.cc:407] Connection header received
0926 03:58:51.352149 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:51.352150 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:51.352157 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:51.352212 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:51.352816 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:51.353311 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:51.354026 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:51.354231 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:51.354295 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:51.357286 (+ 2991us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33472: BlockingRecv error: recv got EOF from 127.0.0.1:33472 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 26 03:58:51 dist-test-slave-nhpw krb5kdc[7862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395131, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 03:58:51.364558 7956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:51.358354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33480 (local address 127.0.98.62:35257)
0926 03:58:51.358401 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:51.358402 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:51.358512 (+ 110us) server_negotiation.cc:407] Connection header received
0926 03:58:51.358535 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:51.358536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:51.358544 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:51.358598 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:51.359006 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:51.359446 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:51.360079 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:51.360245 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:51.360291 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:51.362296 (+ 2005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:51.362297 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:51.362298 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:51.362303 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:51.363652 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:51.363947 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:51.363948 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:51.363949 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:51.363997 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:51.364203 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:51.364204 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:51.364204 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:51.364295 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:51.364366 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:51.364451 (+ 85us) server_negotiation.cc:299] Negotiation successful
0926 03:58:51.364500 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 03:58:51.366022 7906 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33480:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:58:51.367532 7906 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:58:51.373782 8414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:51.369712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35815 (local address 127.0.0.1:43464)
0926 03:58:51.370015 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:51.370026 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:51.370085 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:51.370376 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:51.370379 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:51.370382 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:58:51.370440 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:51.370442 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:51.371413 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:51.371413 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:51.372500 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:51.372503 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:51.373496 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:51.373497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:51.373583 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:51.373595 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 03:58:51.373659 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 03:58:51.373708 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":3,"thread_start_us":81,"threads_started":1}
I20260926 03:58:51.374006 8412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:51.369895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34219 (local address 127.0.0.1:33410)
0926 03:58:51.370205 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:51.370215 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:51.370261 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:51.370581 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:51.370582 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:51.370585 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:58:51.370624 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:51.370625 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:51.371930 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:51.371931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:51.372939 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:51.372941 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:51.373758 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:51.373759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:51.373821 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:51.373825 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 03:58:51.373871 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 03:58:51.373920 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":41,"thread_start_us":87,"threads_started":1}
I20260926 03:58:51.374024 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:51.369942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43464 (local address 127.0.98.3:35815)
0926 03:58:51.370152 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:51.370157 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:51.370167 (+ 10us) server_negotiation.cc:407] Connection header received
0926 03:58:51.370205 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:51.370207 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:51.370222 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:51.370299 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:58:51.370521 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:51.371318 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:51.372569 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:51.373333 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:51.373398 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:51.373465 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:51.373887 (+ 422us) server_negotiation.cc:299] Negotiation successful
0926 03:58:51.373935 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":89,"threads_started":1}
I20260926 03:58:51.374161 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:51.369712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43027 (local address 127.0.0.1:50406)
0926 03:58:51.369851 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:51.369888 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:51.369955 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:51.370491 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:51.370492 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:51.370495 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:58:51.370551 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:51.370553 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:51.371951 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:51.371952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:51.373023 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:51.373026 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:51.373943 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:51.373944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:51.374008 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:51.374012 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 03:58:51.374044 (+ 32us) client_negotiation.cc:240] Negotiation successful
0926 03:58:51.374095 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":45,"threads_started":1}
I20260926 03:58:51.374471 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:51.370214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33410 (local address 127.0.98.2:34219)
0926 03:58:51.370430 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:51.370434 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:51.370443 (+ 9us) server_negotiation.cc:407] Connection header received
0926 03:58:51.370476 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:51.370478 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:51.370492 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:51.370691 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:58:51.370774 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:51.371852 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:51.373059 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:51.373650 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:51.373753 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:51.374277 (+ 524us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:51.374307 (+ 30us) server_negotiation.cc:299] Negotiation successful
0926 03:58:51.374348 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":82,"threads_started":1}
I20260926 03:58:51.374677 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:51.370129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50406 (local address 127.0.98.1:43027)
0926 03:58:51.370321 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:51.370326 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:51.370336 (+ 10us) server_negotiation.cc:407] Connection header received
0926 03:58:51.370372 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:51.370374 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:51.370390 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:51.370638 (+ 248us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:58:51.370701 (+ 63us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:51.371837 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:51.373129 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:51.373848 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:51.374430 (+ 582us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:51.374487 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:51.374506 (+ 19us) server_negotiation.cc:299] Negotiation successful
0926 03:58:51.374567 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":84,"threads_started":1}
I20260926 03:58:51.374853 8324 tablet_service.cc:1467] Processing CreateTablet for tablet 01e97b7c1de541d889150a23683fd8a9 (DEFAULT_TABLE table=test-table [id=c437f68f13224033a2ec8490d22d0016]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:58:51.374984 8184 tablet_service.cc:1467] Processing CreateTablet for tablet 01e97b7c1de541d889150a23683fd8a9 (DEFAULT_TABLE table=test-table [id=c437f68f13224033a2ec8490d22d0016]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:58:51.375222 8324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01e97b7c1de541d889150a23683fd8a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:58:51.375294 8184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01e97b7c1de541d889150a23683fd8a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:58:51.375278 8044 tablet_service.cc:1467] Processing CreateTablet for tablet 01e97b7c1de541d889150a23683fd8a9 (DEFAULT_TABLE table=test-table [id=c437f68f13224033a2ec8490d22d0016]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:58:51.375602 8044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 01e97b7c1de541d889150a23683fd8a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:58:51.378621 8418 tablet_bootstrap.cc:492] T 01e97b7c1de541d889150a23683fd8a9 P cc1f89d3f0cd49588788997ecba18080: Bootstrap starting.
I20260926 03:58:51.378973 8420 tablet_bootstrap.cc:492] T 01e97b7c1de541d889150a23683fd8a9 P bcf2b7fca8ce4133ac3c49f74265078a: Bootstrap starting.
I20260926 03:58:51.378984 8419 tablet_bootstrap.cc:492] T 01e97b7c1de541d889150a23683fd8a9 P 37dfcdbd7b6f4745a9c25c117f83aa74: Bootstrap starting.
I20260926 03:58:51.379370 8418 tablet_bootstrap.cc:654] T 01e97b7c1de541d889150a23683fd8a9 P cc1f89d3f0cd49588788997ecba18080: Neither blocks nor log segments found. Creating new log.
I20260926 03:58:51.379527 8419 tablet_bootstrap.cc:654] T 01e97b7c1de541d889150a23683fd8a9 P 37dfcdbd7b6f4745a9c25c117f83aa74: Neither blocks nor log segments found. Creating new log.
I20260926 03:58:51.379566 8420 tablet_bootstrap.cc:654] T 01e97b7c1de541d889150a23683fd8a9 P bcf2b7fca8ce4133ac3c49f74265078a: Neither blocks nor log segments found. Creating new log.
I20260926 03:58:51.379712 8418 log.cc:824] T 01e97b7c1de541d889150a23683fd8a9 P cc1f89d3f0cd49588788997ecba18080: Log is configured to *not* fsync() on all Append() calls
I20260926 03:58:51.379858 8420 log.cc:824] T 01e97b7c1de541d889150a23683fd8a9 P bcf2b7fca8ce4133ac3c49f74265078a: Log is configured to *not* fsync() on all Append() calls
I20260926 03:58:51.379863 8419 log.cc:824] T 01e97b7c1de541d889150a23683fd8a9 P 37dfcdbd7b6f4745a9c25c117f83aa74: Log is configured to *not* fsync() on all Append() calls
I20260926 03:58:51.380519 8418 tablet_bootstrap.cc:492] T 01e97b7c1de541d889150a23683fd8a9 P cc1f89d3f0cd49588788997ecba18080: No bootstrap required, opened a new log
I20260926 03:58:51.380601 8418 ts_tablet_manager.cc:1397] T 01e97b7c1de541d889150a23683fd8a9 P cc1f89d3f0cd49588788997ecba18080: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 03:58:51.380604 8420 tablet_bootstrap.cc:492] T 01e97b7c1de541d889150a23683fd8a9 P bcf2b7fca8ce4133ac3c49f74265078a: No bootstrap required, opened a new log
I20260926 03:58:51.380687 8420 ts_tablet_manager.cc:1397] T 01e97b7c1de541d889150a23683fd8a9 P bcf2b7fca8ce4133ac3c49f74265078a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 03:58:51.380703 8419 tablet_bootstrap.cc:492] T 01e97b7c1de541d889150a23683fd8a9 P 37dfcdbd7b6f4745a9c25c117f83aa74: No bootstrap required, opened a new log
I20260926 03:58:51.380756 8419 ts_tablet_manager.cc:1397] T 01e97b7c1de541d889150a23683fd8a9 P 37dfcdbd7b6f4745a9c25c117f83aa74: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 03:58:51.382063 8418 raft_consensus.cc:348] T 01e97b7c1de541d889150a23683fd8a9 P cc1f89d3f0cd49588788997ecba18080 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc1f89d3f0cd49588788997ecba18080" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43027 } } peers { permanent_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34219 } } peers { permanent_uuid: "37dfcdbd7b6f4745a9c25c117f83aa74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35815 } }
I20260926 03:58:51.382223 8418 raft_consensus.cc:374] T 01e97b7c1de541d889150a23683fd8a9 P cc1f89d3f0cd49588788997ecba18080 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:58:51.382265 8418 raft_consensus.cc:729] T 01e97b7c1de541d889150a23683fd8a9 P cc1f89d3f0cd49588788997ecba18080 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc1f89d3f0cd49588788997ecba18080, State: Initialized, Role: FOLLOWER
I20260926 03:58:51.382391 8418 consensus_queue.cc:260] T 01e97b7c1de541d889150a23683fd8a9 P cc1f89d3f0cd49588788997ecba18080 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc1f89d3f0cd49588788997ecba18080" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43027 } } peers { permanent_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34219 } } peers { permanent_uuid: "37dfcdbd7b6f4745a9c25c117f83aa74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35815 } }
I20260926 03:58:51.382665 8418 ts_tablet_manager.cc:1428] T 01e97b7c1de541d889150a23683fd8a9 P cc1f89d3f0cd49588788997ecba18080: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 03:58:51.382632 8420 raft_consensus.cc:348] T 01e97b7c1de541d889150a23683fd8a9 P bcf2b7fca8ce4133ac3c49f74265078a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc1f89d3f0cd49588788997ecba18080" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43027 } } peers { permanent_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34219 } } peers { permanent_uuid: "37dfcdbd7b6f4745a9c25c117f83aa74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35815 } }
I20260926 03:58:51.382627 8419 raft_consensus.cc:348] T 01e97b7c1de541d889150a23683fd8a9 P 37dfcdbd7b6f4745a9c25c117f83aa74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc1f89d3f0cd49588788997ecba18080" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43027 } } peers { permanent_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34219 } } peers { permanent_uuid: "37dfcdbd7b6f4745a9c25c117f83aa74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35815 } }
I20260926 03:58:51.382782 8420 raft_consensus.cc:374] T 01e97b7c1de541d889150a23683fd8a9 P bcf2b7fca8ce4133ac3c49f74265078a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:58:51.382782 8419 raft_consensus.cc:374] T 01e97b7c1de541d889150a23683fd8a9 P 37dfcdbd7b6f4745a9c25c117f83aa74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:58:51.382802 8110 heartbeater.cc:503] Master 127.0.98.62:35257 was elected leader, sending a full tablet report...
I20260926 03:58:51.382823 8420 raft_consensus.cc:729] T 01e97b7c1de541d889150a23683fd8a9 P bcf2b7fca8ce4133ac3c49f74265078a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf2b7fca8ce4133ac3c49f74265078a, State: Initialized, Role: FOLLOWER
I20260926 03:58:51.382823 8419 raft_consensus.cc:729] T 01e97b7c1de541d889150a23683fd8a9 P 37dfcdbd7b6f4745a9c25c117f83aa74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37dfcdbd7b6f4745a9c25c117f83aa74, State: Initialized, Role: FOLLOWER
I20260926 03:58:51.382952 8420 consensus_queue.cc:260] T 01e97b7c1de541d889150a23683fd8a9 P bcf2b7fca8ce4133ac3c49f74265078a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc1f89d3f0cd49588788997ecba18080" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43027 } } peers { permanent_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34219 } } peers { permanent_uuid: "37dfcdbd7b6f4745a9c25c117f83aa74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35815 } }
I20260926 03:58:51.382952 8419 consensus_queue.cc:260] T 01e97b7c1de541d889150a23683fd8a9 P 37dfcdbd7b6f4745a9c25c117f83aa74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc1f89d3f0cd49588788997ecba18080" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43027 } } peers { permanent_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34219 } } peers { permanent_uuid: "37dfcdbd7b6f4745a9c25c117f83aa74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35815 } }
I20260926 03:58:51.383188 8420 ts_tablet_manager.cc:1428] T 01e97b7c1de541d889150a23683fd8a9 P bcf2b7fca8ce4133ac3c49f74265078a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 03:58:51.383198 8419 ts_tablet_manager.cc:1428] T 01e97b7c1de541d889150a23683fd8a9 P 37dfcdbd7b6f4745a9c25c117f83aa74: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 03:58:51.383299 8250 heartbeater.cc:503] Master 127.0.98.62:35257 was elected leader, sending a full tablet report...
I20260926 03:58:51.383574 8390 heartbeater.cc:503] Master 127.0.98.62:35257 was elected leader, sending a full tablet report...
W20260926 03:58:51.417341 8251 tablet.cc:2369] T 01e97b7c1de541d889150a23683fd8a9 P bcf2b7fca8ce4133ac3c49f74265078a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:58:51.506676 8111 tablet.cc:2369] T 01e97b7c1de541d889150a23683fd8a9 P cc1f89d3f0cd49588788997ecba18080: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:58:51.535662 8425 raft_consensus.cc:482] T 01e97b7c1de541d889150a23683fd8a9 P bcf2b7fca8ce4133ac3c49f74265078a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:58:51.535785 8425 raft_consensus.cc:504] T 01e97b7c1de541d889150a23683fd8a9 P bcf2b7fca8ce4133ac3c49f74265078a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc1f89d3f0cd49588788997ecba18080" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43027 } } peers { permanent_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34219 } } peers { permanent_uuid: "37dfcdbd7b6f4745a9c25c117f83aa74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35815 } }
I20260926 03:58:51.536099 8425 leader_election.cc:288] T 01e97b7c1de541d889150a23683fd8a9 P bcf2b7fca8ce4133ac3c49f74265078a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc1f89d3f0cd49588788997ecba18080 (127.0.98.1:43027), 37dfcdbd7b6f4745a9c25c117f83aa74 (127.0.98.3:35815)
I20260926 03:58:51.540091 8416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:51.536394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47411 (local address 127.0.98.1:43027)
0926 03:58:51.536440 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:51.536442 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:51.536455 (+ 13us) server_negotiation.cc:407] Connection header received
0926 03:58:51.536526 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:51.536526 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:51.536535 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:51.536646 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:58:51.536839 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:51.537613 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:51.539145 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:51.539728 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:51.539800 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:51.539810 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:51.539981 (+ 171us) server_negotiation.cc:299] Negotiation successful
0926 03:58:51.540022 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 03:58:51.540091 8415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:51.536420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49663 (local address 127.0.98.3:35815)
0926 03:58:51.536460 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:51.536462 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:51.536623 (+ 161us) server_negotiation.cc:407] Connection header received
0926 03:58:51.536674 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:51.536676 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:51.536687 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:51.536739 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:58:51.536896 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:51.537613 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:51.539151 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:51.539720 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:51.539800 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:51.539810 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:51.539981 (+ 171us) server_negotiation.cc:299] Negotiation successful
0926 03:58:51.540022 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 03:58:51.540107 8427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:51.536328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35815 (local address 127.0.98.2:49663)
0926 03:58:51.536514 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:51.536522 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:51.536575 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:51.536760 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:51.536761 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:51.536762 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:58:51.536825 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:51.536826 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:51.537703 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:51.537704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:51.539046 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:51.539049 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:51.539808 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:51.539809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:51.539889 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:51.539893 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 03:58:51.539957 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 03:58:51.540001 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":27,"thread_start_us":106,"threads_started":1}
I20260926 03:58:51.540112 8143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:51.536328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43027 (local address 127.0.98.2:47411)
0926 03:58:51.536384 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:51.536397 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:51.536452 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:51.536667 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:51.536668 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:51.536670 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:58:51.536763 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:51.536765 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:51.537704 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:51.537704 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:51.539046 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:51.539049 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:51.539808 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:51.539809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:51.539892 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:51.539894 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 03:58:51.539957 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 03:58:51.540001 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":100}
I20260926 03:58:51.540563 8344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01e97b7c1de541d889150a23683fd8a9" candidate_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37dfcdbd7b6f4745a9c25c117f83aa74" is_pre_election: true
I20260926 03:58:51.540771 8344 raft_consensus.cc:2454] T 01e97b7c1de541d889150a23683fd8a9 P 37dfcdbd7b6f4745a9c25c117f83aa74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf2b7fca8ce4133ac3c49f74265078a in term 0.
I20260926 03:58:51.540742 8064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01e97b7c1de541d889150a23683fd8a9" candidate_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc1f89d3f0cd49588788997ecba18080" is_pre_election: true
I20260926 03:58:51.540864 8064 raft_consensus.cc:2454] T 01e97b7c1de541d889150a23683fd8a9 P cc1f89d3f0cd49588788997ecba18080 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf2b7fca8ce4133ac3c49f74265078a in term 0.
I20260926 03:58:51.541092 8136 leader_election.cc:302] T 01e97b7c1de541d889150a23683fd8a9 P bcf2b7fca8ce4133ac3c49f74265078a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37dfcdbd7b6f4745a9c25c117f83aa74, bcf2b7fca8ce4133ac3c49f74265078a; no voters:
I20260926 03:58:51.541224 8425 raft_consensus.cc:2781] T 01e97b7c1de541d889150a23683fd8a9 P bcf2b7fca8ce4133ac3c49f74265078a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:58:51.541280 8425 raft_consensus.cc:482] T 01e97b7c1de541d889150a23683fd8a9 P bcf2b7fca8ce4133ac3c49f74265078a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:58:51.541301 8425 raft_consensus.cc:3037] T 01e97b7c1de541d889150a23683fd8a9 P bcf2b7fca8ce4133ac3c49f74265078a [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:58:51.542199 8425 raft_consensus.cc:504] T 01e97b7c1de541d889150a23683fd8a9 P bcf2b7fca8ce4133ac3c49f74265078a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc1f89d3f0cd49588788997ecba18080" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43027 } } peers { permanent_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34219 } } peers { permanent_uuid: "37dfcdbd7b6f4745a9c25c117f83aa74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35815 } }
I20260926 03:58:51.542335 8425 leader_election.cc:288] T 01e97b7c1de541d889150a23683fd8a9 P bcf2b7fca8ce4133ac3c49f74265078a [CANDIDATE]: Term 1 election: Requested vote from peers cc1f89d3f0cd49588788997ecba18080 (127.0.98.1:43027), 37dfcdbd7b6f4745a9c25c117f83aa74 (127.0.98.3:35815)
I20260926 03:58:51.542505 8344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01e97b7c1de541d889150a23683fd8a9" candidate_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37dfcdbd7b6f4745a9c25c117f83aa74"
I20260926 03:58:51.542518 8064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "01e97b7c1de541d889150a23683fd8a9" candidate_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc1f89d3f0cd49588788997ecba18080"
I20260926 03:58:51.542595 8344 raft_consensus.cc:3037] T 01e97b7c1de541d889150a23683fd8a9 P 37dfcdbd7b6f4745a9c25c117f83aa74 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:58:51.542595 8064 raft_consensus.cc:3037] T 01e97b7c1de541d889150a23683fd8a9 P cc1f89d3f0cd49588788997ecba18080 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:58:51.543452 8344 raft_consensus.cc:2454] T 01e97b7c1de541d889150a23683fd8a9 P 37dfcdbd7b6f4745a9c25c117f83aa74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf2b7fca8ce4133ac3c49f74265078a in term 1.
I20260926 03:58:51.543445 8064 raft_consensus.cc:2454] T 01e97b7c1de541d889150a23683fd8a9 P cc1f89d3f0cd49588788997ecba18080 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf2b7fca8ce4133ac3c49f74265078a in term 1.
I20260926 03:58:51.543627 8138 leader_election.cc:302] T 01e97b7c1de541d889150a23683fd8a9 P bcf2b7fca8ce4133ac3c49f74265078a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bcf2b7fca8ce4133ac3c49f74265078a, cc1f89d3f0cd49588788997ecba18080; no voters:
I20260926 03:58:51.543766 8425 raft_consensus.cc:2781] T 01e97b7c1de541d889150a23683fd8a9 P bcf2b7fca8ce4133ac3c49f74265078a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:58:51.543913 8425 raft_consensus.cc:686] T 01e97b7c1de541d889150a23683fd8a9 P bcf2b7fca8ce4133ac3c49f74265078a [term 1 LEADER]: Becoming Leader. State: Replica: bcf2b7fca8ce4133ac3c49f74265078a, State: Running, Role: LEADER
I20260926 03:58:51.544032 8425 consensus_queue.cc:237] T 01e97b7c1de541d889150a23683fd8a9 P bcf2b7fca8ce4133ac3c49f74265078a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc1f89d3f0cd49588788997ecba18080" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43027 } } peers { permanent_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34219 } } peers { permanent_uuid: "37dfcdbd7b6f4745a9c25c117f83aa74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35815 } }
I20260926 03:58:51.544873 7906 catalog_manager.cc:5495] T 01e97b7c1de541d889150a23683fd8a9 P bcf2b7fca8ce4133ac3c49f74265078a reported cstate change: term changed from 0 to 1, leader changed from <none> to bcf2b7fca8ce4133ac3c49f74265078a (127.0.98.2). New cstate: current_term: 1 leader_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc1f89d3f0cd49588788997ecba18080" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34219 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37dfcdbd7b6f4745a9c25c117f83aa74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35815 } health_report { overall_health: UNKNOWN } } }
I20260926 03:58:51.553984 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:51.551736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33422 (local address 127.0.98.2:34219)
0926 03:58:51.551800 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:51.551802 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:51.551809 (+ 7us) server_negotiation.cc:407] Connection header received
0926 03:58:51.551836 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:51.551837 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:51.551851 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:51.551909 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:58:51.552025 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:51.552463 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:51.553256 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:51.553438 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:51.553500 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:51.553686 (+ 186us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:58:51.553789 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:58:51.553860 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:51.553899 (+ 39us) server_negotiation.cc:299] Negotiation successful
0926 03:58:51.553920 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 03:58:51.555668 8064 raft_consensus.cc:1261] T 01e97b7c1de541d889150a23683fd8a9 P cc1f89d3f0cd49588788997ecba18080 [term 1 FOLLOWER]: Refusing update from remote peer bcf2b7fca8ce4133ac3c49f74265078a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:58:51.555678 8344 raft_consensus.cc:1261] T 01e97b7c1de541d889150a23683fd8a9 P 37dfcdbd7b6f4745a9c25c117f83aa74 [term 1 FOLLOWER]: Refusing update from remote peer bcf2b7fca8ce4133ac3c49f74265078a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:58:51.556013 8425 consensus_queue.cc:1035] T 01e97b7c1de541d889150a23683fd8a9 P bcf2b7fca8ce4133ac3c49f74265078a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc1f89d3f0cd49588788997ecba18080" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:58:51.556191 8428 consensus_queue.cc:1035] T 01e97b7c1de541d889150a23683fd8a9 P bcf2b7fca8ce4133ac3c49f74265078a [LEADER]: Connected to new peer: Peer: permanent_uuid: "37dfcdbd7b6f4745a9c25c117f83aa74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:58:51.558954 8433 mvcc.cc:205] Tried to move back new op lower bound from 7333458458849394688 to 7333458458804822016. Current Snapshot: MvccSnapshot[applied={T|T < 7333458458849394688}]
I20260926 03:58:51.559106 8432 mvcc.cc:205] Tried to move back new op lower bound from 7333458458849394688 to 7333458458804822016. Current Snapshot: MvccSnapshot[applied={T|T < 7333458458849394688}]
I20260926 03:58:51.560370 7907 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:58:51.560458 7907 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:58:51.561268 7907 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bac06df8dabe430ca2575d7abbe118ac: Sending DeleteTablet for 3 replicas of tablet 01e97b7c1de541d889150a23683fd8a9
I20260926 03:58:51.561607 8324 tablet_service.cc:1514] Processing DeleteTablet for tablet 01e97b7c1de541d889150a23683fd8a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:58:51 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43464
I20260926 03:58:51.561637 8044 tablet_service.cc:1514] Processing DeleteTablet for tablet 01e97b7c1de541d889150a23683fd8a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:58:51 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50406
I20260926 03:58:51.561651 8184 tablet_service.cc:1514] Processing DeleteTablet for tablet 01e97b7c1de541d889150a23683fd8a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:58:51 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33410
I20260926 03:58:51.561846 8441 tablet_replica.cc:330] T 01e97b7c1de541d889150a23683fd8a9 P 37dfcdbd7b6f4745a9c25c117f83aa74: stopping tablet replica
I20260926 03:58:51.562022 8441 raft_consensus.cc:2229] T 01e97b7c1de541d889150a23683fd8a9 P 37dfcdbd7b6f4745a9c25c117f83aa74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:58:51.562024 7907 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33480:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:58:51.562191 7907 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:58:51.562398 8441 raft_consensus.cc:2258] T 01e97b7c1de541d889150a23683fd8a9 P 37dfcdbd7b6f4745a9c25c117f83aa74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:58:51.563046 8441 ts_tablet_manager.cc:1905] T 01e97b7c1de541d889150a23683fd8a9 P 37dfcdbd7b6f4745a9c25c117f83aa74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:58:51.562408 8442 tablet_replica.cc:330] T 01e97b7c1de541d889150a23683fd8a9 P bcf2b7fca8ce4133ac3c49f74265078a: stopping tablet replica
I20260926 03:58:51.563550 8044 tablet_service.cc:1467] Processing CreateTablet for tablet 86ec3f86d23c4640b465b0bd8b90e038 (DEFAULT_TABLE table=test-table [id=ef84afa9849749449c2543045da196c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:58:51.563686 8044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86ec3f86d23c4640b465b0bd8b90e038. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:58:51.563717 8184 tablet_service.cc:1467] Processing CreateTablet for tablet 86ec3f86d23c4640b465b0bd8b90e038 (DEFAULT_TABLE table=test-table [id=ef84afa9849749449c2543045da196c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:58:51.563813 8184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86ec3f86d23c4640b465b0bd8b90e038. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:58:51.565032 8418 tablet_bootstrap.cc:492] T 86ec3f86d23c4640b465b0bd8b90e038 P cc1f89d3f0cd49588788997ecba18080: Bootstrap starting.
I20260926 03:58:51.565070 8420 tablet_bootstrap.cc:492] T 86ec3f86d23c4640b465b0bd8b90e038 P bcf2b7fca8ce4133ac3c49f74265078a: Bootstrap starting.
I20260926 03:58:51.565240 8324 tablet_service.cc:1467] Processing CreateTablet for tablet 86ec3f86d23c4640b465b0bd8b90e038 (DEFAULT_TABLE table=test-table [id=ef84afa9849749449c2543045da196c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:58:51.565343 8324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 86ec3f86d23c4640b465b0bd8b90e038. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:58:51.565414 8442 raft_consensus.cc:2229] T 01e97b7c1de541d889150a23683fd8a9 P bcf2b7fca8ce4133ac3c49f74265078a [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:58:51.565438 8418 tablet_bootstrap.cc:654] T 86ec3f86d23c4640b465b0bd8b90e038 P cc1f89d3f0cd49588788997ecba18080: Neither blocks nor log segments found. Creating new log.
I20260926 03:58:51.565438 8420 tablet_bootstrap.cc:654] T 86ec3f86d23c4640b465b0bd8b90e038 P bcf2b7fca8ce4133ac3c49f74265078a: Neither blocks nor log segments found. Creating new log.
I20260926 03:58:51.565527 8442 raft_consensus.cc:2258] T 01e97b7c1de541d889150a23683fd8a9 P bcf2b7fca8ce4133ac3c49f74265078a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:58:51.565658 8440 tablet_replica.cc:330] T 01e97b7c1de541d889150a23683fd8a9 P cc1f89d3f0cd49588788997ecba18080: stopping tablet replica
I20260926 03:58:51.565733 8440 raft_consensus.cc:2229] T 01e97b7c1de541d889150a23683fd8a9 P cc1f89d3f0cd49588788997ecba18080 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:58:51.565815 8440 raft_consensus.cc:2258] T 01e97b7c1de541d889150a23683fd8a9 P cc1f89d3f0cd49588788997ecba18080 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:58:51.566252 8440 ts_tablet_manager.cc:1905] T 01e97b7c1de541d889150a23683fd8a9 P cc1f89d3f0cd49588788997ecba18080: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:58:51.566655 8419 tablet_bootstrap.cc:492] T 86ec3f86d23c4640b465b0bd8b90e038 P 37dfcdbd7b6f4745a9c25c117f83aa74: Bootstrap starting.
I20260926 03:58:51.567050 8419 tablet_bootstrap.cc:654] T 86ec3f86d23c4640b465b0bd8b90e038 P 37dfcdbd7b6f4745a9c25c117f83aa74: Neither blocks nor log segments found. Creating new log.
I20260926 03:58:51.567247 8442 ts_tablet_manager.cc:1905] T 01e97b7c1de541d889150a23683fd8a9 P bcf2b7fca8ce4133ac3c49f74265078a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:58:51.568216 8441 ts_tablet_manager.cc:1918] T 01e97b7c1de541d889150a23683fd8a9 P 37dfcdbd7b6f4745a9c25c117f83aa74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:58:51.568334 8441 log.cc:1196] T 01e97b7c1de541d889150a23683fd8a9 P 37dfcdbd7b6f4745a9c25c117f83aa74: Deleting WAL directory at /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/minicluster-data/ts-2/wal/wals/01e97b7c1de541d889150a23683fd8a9
I20260926 03:58:51.568442 8418 tablet_bootstrap.cc:492] T 86ec3f86d23c4640b465b0bd8b90e038 P cc1f89d3f0cd49588788997ecba18080: No bootstrap required, opened a new log
I20260926 03:58:51.568497 8418 ts_tablet_manager.cc:1397] T 86ec3f86d23c4640b465b0bd8b90e038 P cc1f89d3f0cd49588788997ecba18080: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260926 03:58:51.568678 8418 raft_consensus.cc:348] T 86ec3f86d23c4640b465b0bd8b90e038 P cc1f89d3f0cd49588788997ecba18080 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc1f89d3f0cd49588788997ecba18080" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43027 } } peers { permanent_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34219 } } peers { permanent_uuid: "37dfcdbd7b6f4745a9c25c117f83aa74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35815 } }
I20260926 03:58:51.568756 8418 raft_consensus.cc:374] T 86ec3f86d23c4640b465b0bd8b90e038 P cc1f89d3f0cd49588788997ecba18080 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:58:51.568742 8442 ts_tablet_manager.cc:1918] T 01e97b7c1de541d889150a23683fd8a9 P bcf2b7fca8ce4133ac3c49f74265078a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:58:51.568778 8418 raft_consensus.cc:729] T 86ec3f86d23c4640b465b0bd8b90e038 P cc1f89d3f0cd49588788997ecba18080 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc1f89d3f0cd49588788997ecba18080, State: Initialized, Role: FOLLOWER
I20260926 03:58:51.568804 8442 log.cc:1196] T 01e97b7c1de541d889150a23683fd8a9 P bcf2b7fca8ce4133ac3c49f74265078a: Deleting WAL directory at /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/minicluster-data/ts-1/wal/wals/01e97b7c1de541d889150a23683fd8a9
I20260926 03:58:51.568820 8418 consensus_queue.cc:260] T 86ec3f86d23c4640b465b0bd8b90e038 P cc1f89d3f0cd49588788997ecba18080 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc1f89d3f0cd49588788997ecba18080" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43027 } } peers { permanent_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34219 } } peers { permanent_uuid: "37dfcdbd7b6f4745a9c25c117f83aa74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35815 } }
I20260926 03:58:51.568941 8441 ts_tablet_manager.cc:1939] T 01e97b7c1de541d889150a23683fd8a9 P 37dfcdbd7b6f4745a9c25c117f83aa74: Deleting consensus metadata
I20260926 03:58:51.569057 8442 ts_tablet_manager.cc:1939] T 01e97b7c1de541d889150a23683fd8a9 P bcf2b7fca8ce4133ac3c49f74265078a: Deleting consensus metadata
I20260926 03:58:51.569154 8418 ts_tablet_manager.cc:1428] T 86ec3f86d23c4640b465b0bd8b90e038 P cc1f89d3f0cd49588788997ecba18080: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 03:58:51.569459 7892 catalog_manager.cc:4841] TS 37dfcdbd7b6f4745a9c25c117f83aa74 (127.0.98.3:35815): tablet 01e97b7c1de541d889150a23683fd8a9 (table test-table [id=c437f68f13224033a2ec8490d22d0016]) successfully deleted
I20260926 03:58:51.569662 8420 tablet_bootstrap.cc:492] T 86ec3f86d23c4640b465b0bd8b90e038 P bcf2b7fca8ce4133ac3c49f74265078a: No bootstrap required, opened a new log
I20260926 03:58:51.569711 8420 ts_tablet_manager.cc:1397] T 86ec3f86d23c4640b465b0bd8b90e038 P bcf2b7fca8ce4133ac3c49f74265078a: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260926 03:58:51.569849 8420 raft_consensus.cc:348] T 86ec3f86d23c4640b465b0bd8b90e038 P bcf2b7fca8ce4133ac3c49f74265078a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc1f89d3f0cd49588788997ecba18080" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43027 } } peers { permanent_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34219 } } peers { permanent_uuid: "37dfcdbd7b6f4745a9c25c117f83aa74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35815 } }
I20260926 03:58:51.569921 8420 raft_consensus.cc:374] T 86ec3f86d23c4640b465b0bd8b90e038 P bcf2b7fca8ce4133ac3c49f74265078a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:58:51.569940 8420 raft_consensus.cc:729] T 86ec3f86d23c4640b465b0bd8b90e038 P bcf2b7fca8ce4133ac3c49f74265078a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf2b7fca8ce4133ac3c49f74265078a, State: Initialized, Role: FOLLOWER
I20260926 03:58:51.569989 8420 consensus_queue.cc:260] T 86ec3f86d23c4640b465b0bd8b90e038 P bcf2b7fca8ce4133ac3c49f74265078a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc1f89d3f0cd49588788997ecba18080" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43027 } } peers { permanent_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34219 } } peers { permanent_uuid: "37dfcdbd7b6f4745a9c25c117f83aa74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35815 } }
I20260926 03:58:51.570084 8420 ts_tablet_manager.cc:1428] T 86ec3f86d23c4640b465b0bd8b90e038 P bcf2b7fca8ce4133ac3c49f74265078a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 03:58:51.570102 8440 ts_tablet_manager.cc:1918] T 01e97b7c1de541d889150a23683fd8a9 P cc1f89d3f0cd49588788997ecba18080: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:58:51.570180 8440 log.cc:1196] T 01e97b7c1de541d889150a23683fd8a9 P cc1f89d3f0cd49588788997ecba18080: Deleting WAL directory at /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/minicluster-data/ts-0/wal/wals/01e97b7c1de541d889150a23683fd8a9
I20260926 03:58:51.569469 7894 catalog_manager.cc:4841] TS bcf2b7fca8ce4133ac3c49f74265078a (127.0.98.2:34219): tablet 01e97b7c1de541d889150a23683fd8a9 (table test-table [id=c437f68f13224033a2ec8490d22d0016]) successfully deleted
I20260926 03:58:51.570600 8440 ts_tablet_manager.cc:1939] T 01e97b7c1de541d889150a23683fd8a9 P cc1f89d3f0cd49588788997ecba18080: Deleting consensus metadata
I20260926 03:58:51.570940 7894 catalog_manager.cc:4841] TS cc1f89d3f0cd49588788997ecba18080 (127.0.98.1:43027): tablet 01e97b7c1de541d889150a23683fd8a9 (table test-table [id=c437f68f13224033a2ec8490d22d0016]) successfully deleted
I20260926 03:58:51.573028 8419 tablet_bootstrap.cc:492] T 86ec3f86d23c4640b465b0bd8b90e038 P 37dfcdbd7b6f4745a9c25c117f83aa74: No bootstrap required, opened a new log
I20260926 03:58:51.573086 8419 ts_tablet_manager.cc:1397] T 86ec3f86d23c4640b465b0bd8b90e038 P 37dfcdbd7b6f4745a9c25c117f83aa74: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260926 03:58:51.573230 8419 raft_consensus.cc:348] T 86ec3f86d23c4640b465b0bd8b90e038 P 37dfcdbd7b6f4745a9c25c117f83aa74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc1f89d3f0cd49588788997ecba18080" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43027 } } peers { permanent_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34219 } } peers { permanent_uuid: "37dfcdbd7b6f4745a9c25c117f83aa74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35815 } }
I20260926 03:58:51.573298 8419 raft_consensus.cc:374] T 86ec3f86d23c4640b465b0bd8b90e038 P 37dfcdbd7b6f4745a9c25c117f83aa74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:58:51.573319 8419 raft_consensus.cc:729] T 86ec3f86d23c4640b465b0bd8b90e038 P 37dfcdbd7b6f4745a9c25c117f83aa74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37dfcdbd7b6f4745a9c25c117f83aa74, State: Initialized, Role: FOLLOWER
I20260926 03:58:51.573361 8419 consensus_queue.cc:260] T 86ec3f86d23c4640b465b0bd8b90e038 P 37dfcdbd7b6f4745a9c25c117f83aa74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc1f89d3f0cd49588788997ecba18080" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43027 } } peers { permanent_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34219 } } peers { permanent_uuid: "37dfcdbd7b6f4745a9c25c117f83aa74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35815 } }
I20260926 03:58:51.573443 8419 ts_tablet_manager.cc:1428] T 86ec3f86d23c4640b465b0bd8b90e038 P 37dfcdbd7b6f4745a9c25c117f83aa74: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
W20260926 03:58:51.574396 8391 tablet.cc:2369] T 86ec3f86d23c4640b465b0bd8b90e038 P 37dfcdbd7b6f4745a9c25c117f83aa74: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:58:51.696288 8428 raft_consensus.cc:482] T 86ec3f86d23c4640b465b0bd8b90e038 P bcf2b7fca8ce4133ac3c49f74265078a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:58:51.696411 8428 raft_consensus.cc:504] T 86ec3f86d23c4640b465b0bd8b90e038 P bcf2b7fca8ce4133ac3c49f74265078a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc1f89d3f0cd49588788997ecba18080" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43027 } } peers { permanent_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34219 } } peers { permanent_uuid: "37dfcdbd7b6f4745a9c25c117f83aa74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35815 } }
I20260926 03:58:51.696616 8428 leader_election.cc:288] T 86ec3f86d23c4640b465b0bd8b90e038 P bcf2b7fca8ce4133ac3c49f74265078a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc1f89d3f0cd49588788997ecba18080 (127.0.98.1:43027), 37dfcdbd7b6f4745a9c25c117f83aa74 (127.0.98.3:35815)
I20260926 03:58:51.696871 8064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86ec3f86d23c4640b465b0bd8b90e038" candidate_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc1f89d3f0cd49588788997ecba18080" is_pre_election: true
I20260926 03:58:51.696918 8344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86ec3f86d23c4640b465b0bd8b90e038" candidate_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37dfcdbd7b6f4745a9c25c117f83aa74" is_pre_election: true
I20260926 03:58:51.696988 8064 raft_consensus.cc:2454] T 86ec3f86d23c4640b465b0bd8b90e038 P cc1f89d3f0cd49588788997ecba18080 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf2b7fca8ce4133ac3c49f74265078a in term 0.
I20260926 03:58:51.696992 8344 raft_consensus.cc:2454] T 86ec3f86d23c4640b465b0bd8b90e038 P 37dfcdbd7b6f4745a9c25c117f83aa74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf2b7fca8ce4133ac3c49f74265078a in term 0.
I20260926 03:58:51.697150 8138 leader_election.cc:302] T 86ec3f86d23c4640b465b0bd8b90e038 P bcf2b7fca8ce4133ac3c49f74265078a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bcf2b7fca8ce4133ac3c49f74265078a, cc1f89d3f0cd49588788997ecba18080; no voters:
I20260926 03:58:51.697264 8428 raft_consensus.cc:2781] T 86ec3f86d23c4640b465b0bd8b90e038 P bcf2b7fca8ce4133ac3c49f74265078a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:58:51.697331 8428 raft_consensus.cc:482] T 86ec3f86d23c4640b465b0bd8b90e038 P bcf2b7fca8ce4133ac3c49f74265078a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:58:51.697363 8428 raft_consensus.cc:3037] T 86ec3f86d23c4640b465b0bd8b90e038 P bcf2b7fca8ce4133ac3c49f74265078a [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:58:51.698071 8428 raft_consensus.cc:504] T 86ec3f86d23c4640b465b0bd8b90e038 P bcf2b7fca8ce4133ac3c49f74265078a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc1f89d3f0cd49588788997ecba18080" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43027 } } peers { permanent_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34219 } } peers { permanent_uuid: "37dfcdbd7b6f4745a9c25c117f83aa74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35815 } }
I20260926 03:58:51.698208 8428 leader_election.cc:288] T 86ec3f86d23c4640b465b0bd8b90e038 P bcf2b7fca8ce4133ac3c49f74265078a [CANDIDATE]: Term 1 election: Requested vote from peers cc1f89d3f0cd49588788997ecba18080 (127.0.98.1:43027), 37dfcdbd7b6f4745a9c25c117f83aa74 (127.0.98.3:35815)
I20260926 03:58:51.698349 8344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86ec3f86d23c4640b465b0bd8b90e038" candidate_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37dfcdbd7b6f4745a9c25c117f83aa74"
I20260926 03:58:51.698387 8064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "86ec3f86d23c4640b465b0bd8b90e038" candidate_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc1f89d3f0cd49588788997ecba18080"
I20260926 03:58:51.698439 8344 raft_consensus.cc:3037] T 86ec3f86d23c4640b465b0bd8b90e038 P 37dfcdbd7b6f4745a9c25c117f83aa74 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:58:51.698467 8064 raft_consensus.cc:3037] T 86ec3f86d23c4640b465b0bd8b90e038 P cc1f89d3f0cd49588788997ecba18080 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:58:51.699083 8344 raft_consensus.cc:2454] T 86ec3f86d23c4640b465b0bd8b90e038 P 37dfcdbd7b6f4745a9c25c117f83aa74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf2b7fca8ce4133ac3c49f74265078a in term 1.
I20260926 03:58:51.699111 8064 raft_consensus.cc:2454] T 86ec3f86d23c4640b465b0bd8b90e038 P cc1f89d3f0cd49588788997ecba18080 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf2b7fca8ce4133ac3c49f74265078a in term 1.
I20260926 03:58:51.699250 8136 leader_election.cc:302] T 86ec3f86d23c4640b465b0bd8b90e038 P bcf2b7fca8ce4133ac3c49f74265078a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37dfcdbd7b6f4745a9c25c117f83aa74, bcf2b7fca8ce4133ac3c49f74265078a; no voters:
I20260926 03:58:51.699348 8428 raft_consensus.cc:2781] T 86ec3f86d23c4640b465b0bd8b90e038 P bcf2b7fca8ce4133ac3c49f74265078a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:58:51.699435 8428 raft_consensus.cc:686] T 86ec3f86d23c4640b465b0bd8b90e038 P bcf2b7fca8ce4133ac3c49f74265078a [term 1 LEADER]: Becoming Leader. State: Replica: bcf2b7fca8ce4133ac3c49f74265078a, State: Running, Role: LEADER
I20260926 03:58:51.699489 8428 consensus_queue.cc:237] T 86ec3f86d23c4640b465b0bd8b90e038 P bcf2b7fca8ce4133ac3c49f74265078a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc1f89d3f0cd49588788997ecba18080" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43027 } } peers { permanent_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34219 } } peers { permanent_uuid: "37dfcdbd7b6f4745a9c25c117f83aa74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35815 } }
I20260926 03:58:51.700062 7906 catalog_manager.cc:5495] T 86ec3f86d23c4640b465b0bd8b90e038 P bcf2b7fca8ce4133ac3c49f74265078a reported cstate change: term changed from 0 to 1, leader changed from <none> to bcf2b7fca8ce4133ac3c49f74265078a (127.0.98.2). New cstate: current_term: 1 leader_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc1f89d3f0cd49588788997ecba18080" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34219 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37dfcdbd7b6f4745a9c25c117f83aa74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35815 } health_report { overall_health: UNKNOWN } } }
Sep 26 03:58:51 dist-test-slave-nhpw krb5kdc[7862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395130, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 03:58:51.735729 8452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:51.729259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35257 (local address 127.0.0.1:33496)
0926 03:58:51.729433 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:51.729443 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:51.729512 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:51.729627 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:51.729628 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:51.729933 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:51.729986 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:51.729988 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:51.730624 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:51.730625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:51.731189 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:51.731191 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:51.731503 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:51.731504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:51.731544 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:51.732055 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:51.732059 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:51.733454 (+ 1395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:58:51.734802 (+ 1348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:51.734808 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:51.734810 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:51.735019 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:51.735218 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:51.735219 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:51.735219 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:51.735274 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:51.735474 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:58:51.735485 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:58:51.735523 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 03:58:51.735575 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 03:58:51.735660 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":77,"threads_started":1}
I20260926 03:58:51.735831 7956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:51.729330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33496 (local address 127.0.98.62:35257)
0926 03:58:51.729425 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:51.729427 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:51.729508 (+ 81us) server_negotiation.cc:407] Connection header received
0926 03:58:51.729553 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:51.729553 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:51.729561 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:51.729613 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:51.730052 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:51.730547 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:51.731262 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:51.731434 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:51.731485 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:51.733537 (+ 2052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:51.733538 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:51.733538 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:51.733544 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:51.734720 (+ 1176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:51.735102 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:51.735103 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:51.735103 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:51.735156 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:51.735328 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:51.735328 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:51.735329 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:51.735420 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:51.735469 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:51.735628 (+ 159us) server_negotiation.cc:299] Negotiation successful
0926 03:58:51.735709 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":66}
I20260926 03:58:51.737780 7906 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33496:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 03:58:51.737978 7906 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:58:51.739522 8044 tablet_service.cc:1467] Processing CreateTablet for tablet d459ba7347ef4ef5bf947f1390baafe6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c427e4d18f1b452cbe420fd671040993]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 03:58:51.739549 8184 tablet_service.cc:1467] Processing CreateTablet for tablet d459ba7347ef4ef5bf947f1390baafe6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c427e4d18f1b452cbe420fd671040993]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 03:58:51.739697 8044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d459ba7347ef4ef5bf947f1390baafe6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:58:51.739713 8184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d459ba7347ef4ef5bf947f1390baafe6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:58:51.740809 8418 tablet_bootstrap.cc:492] T d459ba7347ef4ef5bf947f1390baafe6 P cc1f89d3f0cd49588788997ecba18080: Bootstrap starting.
I20260926 03:58:51.740810 8420 tablet_bootstrap.cc:492] T d459ba7347ef4ef5bf947f1390baafe6 P bcf2b7fca8ce4133ac3c49f74265078a: Bootstrap starting.
I20260926 03:58:51.741135 8420 tablet_bootstrap.cc:654] T d459ba7347ef4ef5bf947f1390baafe6 P bcf2b7fca8ce4133ac3c49f74265078a: Neither blocks nor log segments found. Creating new log.
I20260926 03:58:51.741209 8418 tablet_bootstrap.cc:654] T d459ba7347ef4ef5bf947f1390baafe6 P cc1f89d3f0cd49588788997ecba18080: Neither blocks nor log segments found. Creating new log.
I20260926 03:58:51.741607 8324 tablet_service.cc:1467] Processing CreateTablet for tablet d459ba7347ef4ef5bf947f1390baafe6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c427e4d18f1b452cbe420fd671040993]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 03:58:51.741677 8420 tablet_bootstrap.cc:492] T d459ba7347ef4ef5bf947f1390baafe6 P bcf2b7fca8ce4133ac3c49f74265078a: No bootstrap required, opened a new log
I20260926 03:58:51.741721 8420 ts_tablet_manager.cc:1397] T d459ba7347ef4ef5bf947f1390baafe6 P bcf2b7fca8ce4133ac3c49f74265078a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 03:58:51.741734 8324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d459ba7347ef4ef5bf947f1390baafe6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:58:51.741838 8420 raft_consensus.cc:348] T d459ba7347ef4ef5bf947f1390baafe6 P bcf2b7fca8ce4133ac3c49f74265078a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc1f89d3f0cd49588788997ecba18080" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43027 } } peers { permanent_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34219 } } peers { permanent_uuid: "37dfcdbd7b6f4745a9c25c117f83aa74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35815 } }
I20260926 03:58:51.741885 8420 raft_consensus.cc:374] T d459ba7347ef4ef5bf947f1390baafe6 P bcf2b7fca8ce4133ac3c49f74265078a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:58:51.741878 8418 tablet_bootstrap.cc:492] T d459ba7347ef4ef5bf947f1390baafe6 P cc1f89d3f0cd49588788997ecba18080: No bootstrap required, opened a new log
I20260926 03:58:51.741899 8420 raft_consensus.cc:729] T d459ba7347ef4ef5bf947f1390baafe6 P bcf2b7fca8ce4133ac3c49f74265078a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf2b7fca8ce4133ac3c49f74265078a, State: Initialized, Role: FOLLOWER
I20260926 03:58:51.741917 8418 ts_tablet_manager.cc:1397] T d459ba7347ef4ef5bf947f1390baafe6 P cc1f89d3f0cd49588788997ecba18080: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 03:58:51.741959 8420 consensus_queue.cc:260] T d459ba7347ef4ef5bf947f1390baafe6 P bcf2b7fca8ce4133ac3c49f74265078a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc1f89d3f0cd49588788997ecba18080" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43027 } } peers { permanent_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34219 } } peers { permanent_uuid: "37dfcdbd7b6f4745a9c25c117f83aa74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35815 } }
I20260926 03:58:51.742057 8420 ts_tablet_manager.cc:1428] T d459ba7347ef4ef5bf947f1390baafe6 P bcf2b7fca8ce4133ac3c49f74265078a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:51.742053 8418 raft_consensus.cc:348] T d459ba7347ef4ef5bf947f1390baafe6 P cc1f89d3f0cd49588788997ecba18080 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc1f89d3f0cd49588788997ecba18080" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43027 } } peers { permanent_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34219 } } peers { permanent_uuid: "37dfcdbd7b6f4745a9c25c117f83aa74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35815 } }
I20260926 03:58:51.742113 8418 raft_consensus.cc:374] T d459ba7347ef4ef5bf947f1390baafe6 P cc1f89d3f0cd49588788997ecba18080 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:58:51.742064 8428 tablet_replica.cc:439] T d459ba7347ef4ef5bf947f1390baafe6 P bcf2b7fca8ce4133ac3c49f74265078a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc1f89d3f0cd49588788997ecba18080" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43027 } } peers { permanent_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34219 } } peers { permanent_uuid: "37dfcdbd7b6f4745a9c25c117f83aa74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35815 } } }
I20260926 03:58:51.742133 8418 raft_consensus.cc:729] T d459ba7347ef4ef5bf947f1390baafe6 P cc1f89d3f0cd49588788997ecba18080 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc1f89d3f0cd49588788997ecba18080, State: Initialized, Role: FOLLOWER
I20260926 03:58:51.742142 8428 tablet_replica.cc:442] T d459ba7347ef4ef5bf947f1390baafe6 P bcf2b7fca8ce4133ac3c49f74265078a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:58:51.742178 8418 consensus_queue.cc:260] T d459ba7347ef4ef5bf947f1390baafe6 P cc1f89d3f0cd49588788997ecba18080 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc1f89d3f0cd49588788997ecba18080" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43027 } } peers { permanent_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34219 } } peers { permanent_uuid: "37dfcdbd7b6f4745a9c25c117f83aa74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35815 } }
I20260926 03:58:51.742257 8418 ts_tablet_manager.cc:1428] T d459ba7347ef4ef5bf947f1390baafe6 P cc1f89d3f0cd49588788997ecba18080: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:51.742317 8424 tablet_replica.cc:439] T d459ba7347ef4ef5bf947f1390baafe6 P cc1f89d3f0cd49588788997ecba18080: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc1f89d3f0cd49588788997ecba18080" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43027 } } peers { permanent_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34219 } } peers { permanent_uuid: "37dfcdbd7b6f4745a9c25c117f83aa74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35815 } } }
I20260926 03:58:51.742408 8424 tablet_replica.cc:442] T d459ba7347ef4ef5bf947f1390baafe6 P cc1f89d3f0cd49588788997ecba18080: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:58:51.742980 8419 tablet_bootstrap.cc:492] T d459ba7347ef4ef5bf947f1390baafe6 P 37dfcdbd7b6f4745a9c25c117f83aa74: Bootstrap starting.
I20260926 03:58:51.743445 8419 tablet_bootstrap.cc:654] T d459ba7347ef4ef5bf947f1390baafe6 P 37dfcdbd7b6f4745a9c25c117f83aa74: Neither blocks nor log segments found. Creating new log.
I20260926 03:58:51.744102 8419 tablet_bootstrap.cc:492] T d459ba7347ef4ef5bf947f1390baafe6 P 37dfcdbd7b6f4745a9c25c117f83aa74: No bootstrap required, opened a new log
I20260926 03:58:51.744143 8419 ts_tablet_manager.cc:1397] T d459ba7347ef4ef5bf947f1390baafe6 P 37dfcdbd7b6f4745a9c25c117f83aa74: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 03:58:51.744253 8419 raft_consensus.cc:348] T d459ba7347ef4ef5bf947f1390baafe6 P 37dfcdbd7b6f4745a9c25c117f83aa74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc1f89d3f0cd49588788997ecba18080" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43027 } } peers { permanent_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34219 } } peers { permanent_uuid: "37dfcdbd7b6f4745a9c25c117f83aa74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35815 } }
I20260926 03:58:51.744302 8419 raft_consensus.cc:374] T d459ba7347ef4ef5bf947f1390baafe6 P 37dfcdbd7b6f4745a9c25c117f83aa74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:58:51.744314 8419 raft_consensus.cc:729] T d459ba7347ef4ef5bf947f1390baafe6 P 37dfcdbd7b6f4745a9c25c117f83aa74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37dfcdbd7b6f4745a9c25c117f83aa74, State: Initialized, Role: FOLLOWER
I20260926 03:58:51.744370 8419 consensus_queue.cc:260] T d459ba7347ef4ef5bf947f1390baafe6 P 37dfcdbd7b6f4745a9c25c117f83aa74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc1f89d3f0cd49588788997ecba18080" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43027 } } peers { permanent_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34219 } } peers { permanent_uuid: "37dfcdbd7b6f4745a9c25c117f83aa74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35815 } }
I20260926 03:58:51.744463 8419 ts_tablet_manager.cc:1428] T d459ba7347ef4ef5bf947f1390baafe6 P 37dfcdbd7b6f4745a9c25c117f83aa74: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:51.744511 8426 tablet_replica.cc:439] T d459ba7347ef4ef5bf947f1390baafe6 P 37dfcdbd7b6f4745a9c25c117f83aa74: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc1f89d3f0cd49588788997ecba18080" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43027 } } peers { permanent_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34219 } } peers { permanent_uuid: "37dfcdbd7b6f4745a9c25c117f83aa74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35815 } } }
I20260926 03:58:51.744611 8426 tablet_replica.cc:442] T d459ba7347ef4ef5bf947f1390baafe6 P 37dfcdbd7b6f4745a9c25c117f83aa74: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:58:51.954883 8428 raft_consensus.cc:482] T d459ba7347ef4ef5bf947f1390baafe6 P bcf2b7fca8ce4133ac3c49f74265078a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:58:51.954974 8428 raft_consensus.cc:504] T d459ba7347ef4ef5bf947f1390baafe6 P bcf2b7fca8ce4133ac3c49f74265078a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc1f89d3f0cd49588788997ecba18080" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43027 } } peers { permanent_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34219 } } peers { permanent_uuid: "37dfcdbd7b6f4745a9c25c117f83aa74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35815 } }
I20260926 03:58:51.955156 8428 leader_election.cc:288] T d459ba7347ef4ef5bf947f1390baafe6 P bcf2b7fca8ce4133ac3c49f74265078a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc1f89d3f0cd49588788997ecba18080 (127.0.98.1:43027), 37dfcdbd7b6f4745a9c25c117f83aa74 (127.0.98.3:35815)
I20260926 03:58:51.955408 8064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d459ba7347ef4ef5bf947f1390baafe6" candidate_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc1f89d3f0cd49588788997ecba18080" is_pre_election: true
I20260926 03:58:51.955428 8344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d459ba7347ef4ef5bf947f1390baafe6" candidate_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37dfcdbd7b6f4745a9c25c117f83aa74" is_pre_election: true
I20260926 03:58:51.955508 8064 raft_consensus.cc:2454] T d459ba7347ef4ef5bf947f1390baafe6 P cc1f89d3f0cd49588788997ecba18080 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf2b7fca8ce4133ac3c49f74265078a in term 0.
I20260926 03:58:51.955515 8344 raft_consensus.cc:2454] T d459ba7347ef4ef5bf947f1390baafe6 P 37dfcdbd7b6f4745a9c25c117f83aa74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf2b7fca8ce4133ac3c49f74265078a in term 0.
I20260926 03:58:51.955708 8136 leader_election.cc:302] T d459ba7347ef4ef5bf947f1390baafe6 P bcf2b7fca8ce4133ac3c49f74265078a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37dfcdbd7b6f4745a9c25c117f83aa74, bcf2b7fca8ce4133ac3c49f74265078a; no voters:
I20260926 03:58:51.955826 8428 raft_consensus.cc:2781] T d459ba7347ef4ef5bf947f1390baafe6 P bcf2b7fca8ce4133ac3c49f74265078a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:58:51.955870 8428 raft_consensus.cc:482] T d459ba7347ef4ef5bf947f1390baafe6 P bcf2b7fca8ce4133ac3c49f74265078a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:58:51.955894 8428 raft_consensus.cc:3037] T d459ba7347ef4ef5bf947f1390baafe6 P bcf2b7fca8ce4133ac3c49f74265078a [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:58:51.956507 8428 raft_consensus.cc:504] T d459ba7347ef4ef5bf947f1390baafe6 P bcf2b7fca8ce4133ac3c49f74265078a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc1f89d3f0cd49588788997ecba18080" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43027 } } peers { permanent_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34219 } } peers { permanent_uuid: "37dfcdbd7b6f4745a9c25c117f83aa74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35815 } }
I20260926 03:58:51.956665 8428 leader_election.cc:288] T d459ba7347ef4ef5bf947f1390baafe6 P bcf2b7fca8ce4133ac3c49f74265078a [CANDIDATE]: Term 1 election: Requested vote from peers cc1f89d3f0cd49588788997ecba18080 (127.0.98.1:43027), 37dfcdbd7b6f4745a9c25c117f83aa74 (127.0.98.3:35815)
I20260926 03:58:51.956779 8064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d459ba7347ef4ef5bf947f1390baafe6" candidate_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc1f89d3f0cd49588788997ecba18080"
I20260926 03:58:51.956835 8344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d459ba7347ef4ef5bf947f1390baafe6" candidate_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37dfcdbd7b6f4745a9c25c117f83aa74"
I20260926 03:58:51.956883 8064 raft_consensus.cc:3037] T d459ba7347ef4ef5bf947f1390baafe6 P cc1f89d3f0cd49588788997ecba18080 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:58:51.956904 8344 raft_consensus.cc:3037] T d459ba7347ef4ef5bf947f1390baafe6 P 37dfcdbd7b6f4745a9c25c117f83aa74 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:58:51.957605 8064 raft_consensus.cc:2454] T d459ba7347ef4ef5bf947f1390baafe6 P cc1f89d3f0cd49588788997ecba18080 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf2b7fca8ce4133ac3c49f74265078a in term 1.
I20260926 03:58:51.957613 8344 raft_consensus.cc:2454] T d459ba7347ef4ef5bf947f1390baafe6 P 37dfcdbd7b6f4745a9c25c117f83aa74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf2b7fca8ce4133ac3c49f74265078a in term 1.
I20260926 03:58:51.957778 8136 leader_election.cc:302] T d459ba7347ef4ef5bf947f1390baafe6 P bcf2b7fca8ce4133ac3c49f74265078a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37dfcdbd7b6f4745a9c25c117f83aa74, bcf2b7fca8ce4133ac3c49f74265078a; no voters:
I20260926 03:58:51.957875 8428 raft_consensus.cc:2781] T d459ba7347ef4ef5bf947f1390baafe6 P bcf2b7fca8ce4133ac3c49f74265078a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:58:51.957955 8428 raft_consensus.cc:686] T d459ba7347ef4ef5bf947f1390baafe6 P bcf2b7fca8ce4133ac3c49f74265078a [term 1 LEADER]: Becoming Leader. State: Replica: bcf2b7fca8ce4133ac3c49f74265078a, State: Running, Role: LEADER
I20260926 03:58:51.958009 8428 consensus_queue.cc:237] T d459ba7347ef4ef5bf947f1390baafe6 P bcf2b7fca8ce4133ac3c49f74265078a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc1f89d3f0cd49588788997ecba18080" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43027 } } peers { permanent_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34219 } } peers { permanent_uuid: "37dfcdbd7b6f4745a9c25c117f83aa74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35815 } }
I20260926 03:58:51.958338 8425 tablet_replica.cc:439] T d459ba7347ef4ef5bf947f1390baafe6 P bcf2b7fca8ce4133ac3c49f74265078a: TxnStatusTablet state changed. Reason: New leader bcf2b7fca8ce4133ac3c49f74265078a. Latest consensus state: current_term: 1 leader_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc1f89d3f0cd49588788997ecba18080" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43027 } } peers { permanent_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34219 } } peers { permanent_uuid: "37dfcdbd7b6f4745a9c25c117f83aa74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35815 } } }
I20260926 03:58:51.958434 8425 tablet_replica.cc:442] T d459ba7347ef4ef5bf947f1390baafe6 P bcf2b7fca8ce4133ac3c49f74265078a: This TxnStatusTablet replica's current role is: LEADER
I20260926 03:58:51.958599 8457 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 03:58:51.958742 8457 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 03:58:51.958640 7906 catalog_manager.cc:5495] T d459ba7347ef4ef5bf947f1390baafe6 P bcf2b7fca8ce4133ac3c49f74265078a reported cstate change: term changed from 0 to 1, leader changed from <none> to bcf2b7fca8ce4133ac3c49f74265078a (127.0.98.2). New cstate: current_term: 1 leader_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc1f89d3f0cd49588788997ecba18080" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34219 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37dfcdbd7b6f4745a9c25c117f83aa74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35815 } health_report { overall_health: UNKNOWN } } }
I20260926 03:58:51.972621 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:51.970363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33432 (local address 127.0.98.2:34219)
0926 03:58:51.970410 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:51.970412 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:51.970440 (+ 28us) server_negotiation.cc:407] Connection header received
0926 03:58:51.970517 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:51.970519 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:51.970529 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:51.970592 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:58:51.970704 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:51.971164 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:51.971906 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:51.972098 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:51.972182 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:51.972361 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:58:51.972401 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:58:51.972463 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:51.972528 (+ 65us) server_negotiation.cc:299] Negotiation successful
0926 03:58:51.972547 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 03:58:51.972657 8452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:51.970335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34219 (local address 127.0.0.1:33432)
0926 03:58:51.970391 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:51.970403 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:51.970460 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:51.970599 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:51.970600 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:51.970601 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 03:58:51.970646 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:51.970648 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:51.971246 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:51.971247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:51.971818 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:51.971820 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:51.972184 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:51.972185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:51.972248 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:51.972270 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 03:58:51.972463 (+ 193us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 03:58:51.972471 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 03:58:51.972523 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 03:58:51.972569 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17}
I20260926 03:58:51.973507 8344 raft_consensus.cc:1261] T d459ba7347ef4ef5bf947f1390baafe6 P 37dfcdbd7b6f4745a9c25c117f83aa74 [term 1 FOLLOWER]: Refusing update from remote peer bcf2b7fca8ce4133ac3c49f74265078a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:58:51.973507 8064 raft_consensus.cc:1261] T d459ba7347ef4ef5bf947f1390baafe6 P cc1f89d3f0cd49588788997ecba18080 [term 1 FOLLOWER]: Refusing update from remote peer bcf2b7fca8ce4133ac3c49f74265078a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:58:51.973709 8425 consensus_queue.cc:1035] T d459ba7347ef4ef5bf947f1390baafe6 P bcf2b7fca8ce4133ac3c49f74265078a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc1f89d3f0cd49588788997ecba18080" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:58:51.973851 8428 consensus_queue.cc:1035] T d459ba7347ef4ef5bf947f1390baafe6 P bcf2b7fca8ce4133ac3c49f74265078a [LEADER]: Connected to new peer: Peer: permanent_uuid: "37dfcdbd7b6f4745a9c25c117f83aa74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:58:51.974192 8424 tablet_replica.cc:439] T d459ba7347ef4ef5bf947f1390baafe6 P cc1f89d3f0cd49588788997ecba18080: TxnStatusTablet state changed. Reason: New leader bcf2b7fca8ce4133ac3c49f74265078a. Latest consensus state: current_term: 1 leader_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc1f89d3f0cd49588788997ecba18080" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43027 } } peers { permanent_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34219 } } peers { permanent_uuid: "37dfcdbd7b6f4745a9c25c117f83aa74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35815 } } }
I20260926 03:58:51.974282 8424 tablet_replica.cc:442] T d459ba7347ef4ef5bf947f1390baafe6 P cc1f89d3f0cd49588788997ecba18080: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:58:51.974859 8426 tablet_replica.cc:439] T d459ba7347ef4ef5bf947f1390baafe6 P 37dfcdbd7b6f4745a9c25c117f83aa74: TxnStatusTablet state changed. Reason: New leader bcf2b7fca8ce4133ac3c49f74265078a. Latest consensus state: current_term: 1 leader_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc1f89d3f0cd49588788997ecba18080" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43027 } } peers { permanent_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34219 } } peers { permanent_uuid: "37dfcdbd7b6f4745a9c25c117f83aa74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35815 } } }
I20260926 03:58:51.974952 8426 tablet_replica.cc:442] T d459ba7347ef4ef5bf947f1390baafe6 P 37dfcdbd7b6f4745a9c25c117f83aa74: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:58:51.975631 8428 tablet_replica.cc:439] T d459ba7347ef4ef5bf947f1390baafe6 P bcf2b7fca8ce4133ac3c49f74265078a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc1f89d3f0cd49588788997ecba18080" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43027 } } peers { permanent_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34219 } } peers { permanent_uuid: "37dfcdbd7b6f4745a9c25c117f83aa74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35815 } } }
I20260926 03:58:51.975704 8428 tablet_replica.cc:442] T d459ba7347ef4ef5bf947f1390baafe6 P bcf2b7fca8ce4133ac3c49f74265078a: This TxnStatusTablet replica's current role is: LEADER
I20260926 03:58:51.975944 8425 tablet_replica.cc:439] T d459ba7347ef4ef5bf947f1390baafe6 P bcf2b7fca8ce4133ac3c49f74265078a: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc1f89d3f0cd49588788997ecba18080" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43027 } } peers { permanent_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34219 } } peers { permanent_uuid: "37dfcdbd7b6f4745a9c25c117f83aa74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35815 } } }
I20260926 03:58:51.976033 8425 tablet_replica.cc:442] T d459ba7347ef4ef5bf947f1390baafe6 P bcf2b7fca8ce4133ac3c49f74265078a: This TxnStatusTablet replica's current role is: LEADER
I20260926 03:58:51.976280 8431 mvcc.cc:205] Tried to move back new op lower bound from 7333458460562018304 to 7333458460500541440. Current Snapshot: MvccSnapshot[applied={T|T < 7333458460562018304 or (T in {7333458460562018304})}]
I20260926 03:58:51.976375 8424 tablet_replica.cc:439] T d459ba7347ef4ef5bf947f1390baafe6 P cc1f89d3f0cd49588788997ecba18080: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc1f89d3f0cd49588788997ecba18080" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43027 } } peers { permanent_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34219 } } peers { permanent_uuid: "37dfcdbd7b6f4745a9c25c117f83aa74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35815 } } }
I20260926 03:58:51.976459 8424 tablet_replica.cc:442] T d459ba7347ef4ef5bf947f1390baafe6 P cc1f89d3f0cd49588788997ecba18080: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:58:51.976881 8426 tablet_replica.cc:439] T d459ba7347ef4ef5bf947f1390baafe6 P 37dfcdbd7b6f4745a9c25c117f83aa74: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc1f89d3f0cd49588788997ecba18080" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43027 } } peers { permanent_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34219 } } peers { permanent_uuid: "37dfcdbd7b6f4745a9c25c117f83aa74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35815 } } }
I20260926 03:58:51.976955 8426 tablet_replica.cc:442] T d459ba7347ef4ef5bf947f1390baafe6 P 37dfcdbd7b6f4745a9c25c117f83aa74: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 03:58:51.981738 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:51.979141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44069 (local address 127.0.98.2:34219)
0926 03:58:51.979199 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:51.979201 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:51.979331 (+ 130us) server_negotiation.cc:407] Connection header received
0926 03:58:51.979382 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:51.979383 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:51.979390 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:51.979428 (+ 38us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:58:51.979536 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:51.979967 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:51.981104 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:51.981279 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:51.981327 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:51.981511 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:58:51.981538 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:58:51.981579 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:51.981664 (+ 85us) server_negotiation.cc:299] Negotiation successful
0926 03:58:51.981678 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 03:58:51.981820 8465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:51.979093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34219 (local address 127.0.98.2:44069)
0926 03:58:51.979242 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:51.979265 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:51.979327 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:51.979440 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:51.979440 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:51.979442 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 03:58:51.979484 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:51.979485 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:51.980039 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:51.980040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:51.981013 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:51.981016 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:51.981351 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:51.981352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:51.981412 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:51.981426 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 03:58:51.981607 (+ 181us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 03:58:51.981609 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 03:58:51.981663 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 03:58:51.981704 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":91,"threads_started":1}
I20260926 03:58:51.983911 8344 raft_consensus.cc:1261] T 86ec3f86d23c4640b465b0bd8b90e038 P 37dfcdbd7b6f4745a9c25c117f83aa74 [term 1 FOLLOWER]: Refusing update from remote peer bcf2b7fca8ce4133ac3c49f74265078a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:58:51.983911 8064 raft_consensus.cc:1261] T 86ec3f86d23c4640b465b0bd8b90e038 P cc1f89d3f0cd49588788997ecba18080 [term 1 FOLLOWER]: Refusing update from remote peer bcf2b7fca8ce4133ac3c49f74265078a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:58:51.984122 8462 consensus_queue.cc:1035] T 86ec3f86d23c4640b465b0bd8b90e038 P bcf2b7fca8ce4133ac3c49f74265078a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc1f89d3f0cd49588788997ecba18080" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:58:51.984257 8466 consensus_queue.cc:1035] T 86ec3f86d23c4640b465b0bd8b90e038 P bcf2b7fca8ce4133ac3c49f74265078a [LEADER]: Connected to new peer: Peer: permanent_uuid: "37dfcdbd7b6f4745a9c25c117f83aa74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:58:51.999444 7906 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:58:51.999564 7906 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:33480:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:58:52.000156 7906 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bac06df8dabe430ca2575d7abbe118ac: Sending DeleteTablet for 3 replicas of tablet 86ec3f86d23c4640b465b0bd8b90e038
I20260926 03:58:52.000384 8324 tablet_service.cc:1514] Processing DeleteTablet for tablet 86ec3f86d23c4640b465b0bd8b90e038 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:58:51 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43464
I20260926 03:58:52.000442 8044 tablet_service.cc:1514] Processing DeleteTablet for tablet 86ec3f86d23c4640b465b0bd8b90e038 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:58:51 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50406
I20260926 03:58:52.000470 8441 tablet_replica.cc:330] T 86ec3f86d23c4640b465b0bd8b90e038 P 37dfcdbd7b6f4745a9c25c117f83aa74: stopping tablet replica
I20260926 03:58:52.000514 8441 raft_consensus.cc:2229] T 86ec3f86d23c4640b465b0bd8b90e038 P 37dfcdbd7b6f4745a9c25c117f83aa74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:58:52.000546 8441 raft_consensus.cc:2258] T 86ec3f86d23c4640b465b0bd8b90e038 P 37dfcdbd7b6f4745a9c25c117f83aa74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:58:52.000538 8183 tablet_service.cc:1514] Processing DeleteTablet for tablet 86ec3f86d23c4640b465b0bd8b90e038 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:58:51 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33410
I20260926 03:58:52.000541 8440 tablet_replica.cc:330] T 86ec3f86d23c4640b465b0bd8b90e038 P cc1f89d3f0cd49588788997ecba18080: stopping tablet replica
I20260926 03:58:52.000607 8440 raft_consensus.cc:2229] T 86ec3f86d23c4640b465b0bd8b90e038 P cc1f89d3f0cd49588788997ecba18080 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:58:52.000609 8442 tablet_replica.cc:330] T 86ec3f86d23c4640b465b0bd8b90e038 P bcf2b7fca8ce4133ac3c49f74265078a: stopping tablet replica
I20260926 03:58:52.000660 8440 raft_consensus.cc:2258] T 86ec3f86d23c4640b465b0bd8b90e038 P cc1f89d3f0cd49588788997ecba18080 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:58:52.000677 8442 raft_consensus.cc:2229] T 86ec3f86d23c4640b465b0bd8b90e038 P bcf2b7fca8ce4133ac3c49f74265078a [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:58:52.000730 8442 raft_consensus.cc:2258] T 86ec3f86d23c4640b465b0bd8b90e038 P bcf2b7fca8ce4133ac3c49f74265078a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:58:52.000981 8442 ts_tablet_manager.cc:1905] T 86ec3f86d23c4640b465b0bd8b90e038 P bcf2b7fca8ce4133ac3c49f74265078a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:58:52.002177 8441 ts_tablet_manager.cc:1905] T 86ec3f86d23c4640b465b0bd8b90e038 P 37dfcdbd7b6f4745a9c25c117f83aa74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:58:52.002696 8442 ts_tablet_manager.cc:1918] T 86ec3f86d23c4640b465b0bd8b90e038 P bcf2b7fca8ce4133ac3c49f74265078a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 03:58:52.002743 8442 log.cc:1196] T 86ec3f86d23c4640b465b0bd8b90e038 P bcf2b7fca8ce4133ac3c49f74265078a: Deleting WAL directory at /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/minicluster-data/ts-1/wal/wals/86ec3f86d23c4640b465b0bd8b90e038
I20260926 03:58:52.002951 8442 ts_tablet_manager.cc:1939] T 86ec3f86d23c4640b465b0bd8b90e038 P bcf2b7fca8ce4133ac3c49f74265078a: Deleting consensus metadata
I20260926 03:58:52.003393 7894 catalog_manager.cc:4841] TS bcf2b7fca8ce4133ac3c49f74265078a (127.0.98.2:34219): tablet 86ec3f86d23c4640b465b0bd8b90e038 (table test-table [id=ef84afa9849749449c2543045da196c7]) successfully deleted
I20260926 03:58:52.003782 8441 ts_tablet_manager.cc:1918] T 86ec3f86d23c4640b465b0bd8b90e038 P 37dfcdbd7b6f4745a9c25c117f83aa74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 03:58:52.003819 8441 log.cc:1196] T 86ec3f86d23c4640b465b0bd8b90e038 P 37dfcdbd7b6f4745a9c25c117f83aa74: Deleting WAL directory at /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/minicluster-data/ts-2/wal/wals/86ec3f86d23c4640b465b0bd8b90e038
I20260926 03:58:52.003964 8441 ts_tablet_manager.cc:1939] T 86ec3f86d23c4640b465b0bd8b90e038 P 37dfcdbd7b6f4745a9c25c117f83aa74: Deleting consensus metadata
I20260926 03:58:52.004488 7892 catalog_manager.cc:4841] TS 37dfcdbd7b6f4745a9c25c117f83aa74 (127.0.98.3:35815): tablet 86ec3f86d23c4640b465b0bd8b90e038 (table test-table [id=ef84afa9849749449c2543045da196c7]) successfully deleted
I20260926 03:58:52.004824 8440 ts_tablet_manager.cc:1905] T 86ec3f86d23c4640b465b0bd8b90e038 P cc1f89d3f0cd49588788997ecba18080: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:58:52.006664 8440 ts_tablet_manager.cc:1918] T 86ec3f86d23c4640b465b0bd8b90e038 P cc1f89d3f0cd49588788997ecba18080: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 03:58:52.006727 8440 log.cc:1196] T 86ec3f86d23c4640b465b0bd8b90e038 P cc1f89d3f0cd49588788997ecba18080: Deleting WAL directory at /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/minicluster-data/ts-0/wal/wals/86ec3f86d23c4640b465b0bd8b90e038
I20260926 03:58:52.006949 8440 ts_tablet_manager.cc:1939] T 86ec3f86d23c4640b465b0bd8b90e038 P cc1f89d3f0cd49588788997ecba18080: Deleting consensus metadata
I20260926 03:58:52.007453 7894 catalog_manager.cc:4841] TS cc1f89d3f0cd49588788997ecba18080 (127.0.98.1:43027): tablet 86ec3f86d23c4640b465b0bd8b90e038 (table test-table [id=ef84afa9849749449c2543045da196c7]) successfully deleted
Sep 26 03:58:52 dist-test-slave-nhpw krb5kdc[7862](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 03:58:52 dist-test-slave-nhpw krb5kdc[7862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:58:52 dist-test-slave-nhpw krb5kdc[7862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 03:58:52.025576 7956 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:58:52.019397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33500 (local address 127.0.98.62:35257)
0926 03:58:52.019457 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:52.019459 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:52.019593 (+ 134us) server_negotiation.cc:407] Connection header received
0926 03:58:52.019665 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:52.019665 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:52.019675 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:52.019718 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:52.020388 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:52.020946 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:52.022048 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:52.022288 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:52.022359 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:52.025505 (+ 3146us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33500: BlockingRecv error: recv got EOF from 127.0.0.1:33500 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 26 03:58:52 dist-test-slave-nhpw krb5kdc[7862](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395132, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260926 03:58:52.033639 7956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:52.026374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33510 (local address 127.0.98.62:35257)
0926 03:58:52.026416 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:52.026418 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:52.026522 (+ 104us) server_negotiation.cc:407] Connection header received
0926 03:58:52.026595 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:52.026596 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:52.026606 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:52.026654 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:52.027189 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:52.027865 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:52.028548 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:52.028853 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:52.028939 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:52.031045 (+ 2106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:52.031046 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:52.031047 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:52.031054 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:52.032469 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:52.032864 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:52.032865 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:52.032866 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:52.032923 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:52.033133 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:52.033134 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:52.033134 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:52.033239 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:52.033347 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:52.033455 (+ 108us) server_negotiation.cc:299] Negotiation successful
0926 03:58:52.033532 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 03:58:52.035076 7906 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33510:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:58:52.035223 7906 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:58:52.036686 8044 tablet_service.cc:1467] Processing CreateTablet for tablet 4b5fcb2756ae4ac5bcfd97ad5ccb464f (DEFAULT_TABLE table=test-table [id=f4a08b560cab4c47b477933f0fb2646d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:58:52.036702 8183 tablet_service.cc:1467] Processing CreateTablet for tablet 4b5fcb2756ae4ac5bcfd97ad5ccb464f (DEFAULT_TABLE table=test-table [id=f4a08b560cab4c47b477933f0fb2646d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:58:52.036702 8324 tablet_service.cc:1467] Processing CreateTablet for tablet 4b5fcb2756ae4ac5bcfd97ad5ccb464f (DEFAULT_TABLE table=test-table [id=f4a08b560cab4c47b477933f0fb2646d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:58:52.036820 8324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b5fcb2756ae4ac5bcfd97ad5ccb464f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:58:52.036821 8044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b5fcb2756ae4ac5bcfd97ad5ccb464f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:58:52.036821 8183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b5fcb2756ae4ac5bcfd97ad5ccb464f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:58:52.038230 8419 tablet_bootstrap.cc:492] T 4b5fcb2756ae4ac5bcfd97ad5ccb464f P 37dfcdbd7b6f4745a9c25c117f83aa74: Bootstrap starting.
I20260926 03:58:52.038463 8418 tablet_bootstrap.cc:492] T 4b5fcb2756ae4ac5bcfd97ad5ccb464f P cc1f89d3f0cd49588788997ecba18080: Bootstrap starting.
I20260926 03:58:52.038676 8419 tablet_bootstrap.cc:654] T 4b5fcb2756ae4ac5bcfd97ad5ccb464f P 37dfcdbd7b6f4745a9c25c117f83aa74: Neither blocks nor log segments found. Creating new log.
I20260926 03:58:52.038813 8418 tablet_bootstrap.cc:654] T 4b5fcb2756ae4ac5bcfd97ad5ccb464f P cc1f89d3f0cd49588788997ecba18080: Neither blocks nor log segments found. Creating new log.
I20260926 03:58:52.038960 8420 tablet_bootstrap.cc:492] T 4b5fcb2756ae4ac5bcfd97ad5ccb464f P bcf2b7fca8ce4133ac3c49f74265078a: Bootstrap starting.
I20260926 03:58:52.039283 8420 tablet_bootstrap.cc:654] T 4b5fcb2756ae4ac5bcfd97ad5ccb464f P bcf2b7fca8ce4133ac3c49f74265078a: Neither blocks nor log segments found. Creating new log.
I20260926 03:58:52.039613 8418 tablet_bootstrap.cc:492] T 4b5fcb2756ae4ac5bcfd97ad5ccb464f P cc1f89d3f0cd49588788997ecba18080: No bootstrap required, opened a new log
I20260926 03:58:52.039659 8418 ts_tablet_manager.cc:1397] T 4b5fcb2756ae4ac5bcfd97ad5ccb464f P cc1f89d3f0cd49588788997ecba18080: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260926 03:58:52.039810 8419 tablet_bootstrap.cc:492] T 4b5fcb2756ae4ac5bcfd97ad5ccb464f P 37dfcdbd7b6f4745a9c25c117f83aa74: No bootstrap required, opened a new log
I20260926 03:58:52.039790 8418 raft_consensus.cc:348] T 4b5fcb2756ae4ac5bcfd97ad5ccb464f P cc1f89d3f0cd49588788997ecba18080 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc1f89d3f0cd49588788997ecba18080" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43027 } } peers { permanent_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34219 } } peers { permanent_uuid: "37dfcdbd7b6f4745a9c25c117f83aa74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35815 } }
I20260926 03:58:52.039844 8419 ts_tablet_manager.cc:1397] T 4b5fcb2756ae4ac5bcfd97ad5ccb464f P 37dfcdbd7b6f4745a9c25c117f83aa74: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 03:58:52.039862 8418 raft_consensus.cc:374] T 4b5fcb2756ae4ac5bcfd97ad5ccb464f P cc1f89d3f0cd49588788997ecba18080 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:58:52.039884 8418 raft_consensus.cc:729] T 4b5fcb2756ae4ac5bcfd97ad5ccb464f P cc1f89d3f0cd49588788997ecba18080 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc1f89d3f0cd49588788997ecba18080, State: Initialized, Role: FOLLOWER
I20260926 03:58:52.039934 8418 consensus_queue.cc:260] T 4b5fcb2756ae4ac5bcfd97ad5ccb464f P cc1f89d3f0cd49588788997ecba18080 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc1f89d3f0cd49588788997ecba18080" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43027 } } peers { permanent_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34219 } } peers { permanent_uuid: "37dfcdbd7b6f4745a9c25c117f83aa74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35815 } }
I20260926 03:58:52.040032 8418 ts_tablet_manager.cc:1428] T 4b5fcb2756ae4ac5bcfd97ad5ccb464f P cc1f89d3f0cd49588788997ecba18080: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:52.040002 8419 raft_consensus.cc:348] T 4b5fcb2756ae4ac5bcfd97ad5ccb464f P 37dfcdbd7b6f4745a9c25c117f83aa74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc1f89d3f0cd49588788997ecba18080" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43027 } } peers { permanent_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34219 } } peers { permanent_uuid: "37dfcdbd7b6f4745a9c25c117f83aa74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35815 } }
I20260926 03:58:52.040066 8419 raft_consensus.cc:374] T 4b5fcb2756ae4ac5bcfd97ad5ccb464f P 37dfcdbd7b6f4745a9c25c117f83aa74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:58:52.040102 8419 raft_consensus.cc:729] T 4b5fcb2756ae4ac5bcfd97ad5ccb464f P 37dfcdbd7b6f4745a9c25c117f83aa74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37dfcdbd7b6f4745a9c25c117f83aa74, State: Initialized, Role: FOLLOWER
I20260926 03:58:52.040143 8419 consensus_queue.cc:260] T 4b5fcb2756ae4ac5bcfd97ad5ccb464f P 37dfcdbd7b6f4745a9c25c117f83aa74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc1f89d3f0cd49588788997ecba18080" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43027 } } peers { permanent_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34219 } } peers { permanent_uuid: "37dfcdbd7b6f4745a9c25c117f83aa74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35815 } }
I20260926 03:58:52.040226 8419 ts_tablet_manager.cc:1428] T 4b5fcb2756ae4ac5bcfd97ad5ccb464f P 37dfcdbd7b6f4745a9c25c117f83aa74: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:52.040745 8420 tablet_bootstrap.cc:492] T 4b5fcb2756ae4ac5bcfd97ad5ccb464f P bcf2b7fca8ce4133ac3c49f74265078a: No bootstrap required, opened a new log
I20260926 03:58:52.040793 8420 ts_tablet_manager.cc:1397] T 4b5fcb2756ae4ac5bcfd97ad5ccb464f P bcf2b7fca8ce4133ac3c49f74265078a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 03:58:52.040949 8420 raft_consensus.cc:348] T 4b5fcb2756ae4ac5bcfd97ad5ccb464f P bcf2b7fca8ce4133ac3c49f74265078a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc1f89d3f0cd49588788997ecba18080" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43027 } } peers { permanent_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34219 } } peers { permanent_uuid: "37dfcdbd7b6f4745a9c25c117f83aa74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35815 } }
I20260926 03:58:52.041034 8420 raft_consensus.cc:374] T 4b5fcb2756ae4ac5bcfd97ad5ccb464f P bcf2b7fca8ce4133ac3c49f74265078a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:58:52.041055 8420 raft_consensus.cc:729] T 4b5fcb2756ae4ac5bcfd97ad5ccb464f P bcf2b7fca8ce4133ac3c49f74265078a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf2b7fca8ce4133ac3c49f74265078a, State: Initialized, Role: FOLLOWER
I20260926 03:58:52.041110 8420 consensus_queue.cc:260] T 4b5fcb2756ae4ac5bcfd97ad5ccb464f P bcf2b7fca8ce4133ac3c49f74265078a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc1f89d3f0cd49588788997ecba18080" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43027 } } peers { permanent_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34219 } } peers { permanent_uuid: "37dfcdbd7b6f4745a9c25c117f83aa74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35815 } }
I20260926 03:58:52.041206 8420 ts_tablet_manager.cc:1428] T 4b5fcb2756ae4ac5bcfd97ad5ccb464f P bcf2b7fca8ce4133ac3c49f74265078a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:52.103480 8424 raft_consensus.cc:482] T 4b5fcb2756ae4ac5bcfd97ad5ccb464f P cc1f89d3f0cd49588788997ecba18080 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:58:52.103583 8424 raft_consensus.cc:504] T 4b5fcb2756ae4ac5bcfd97ad5ccb464f P cc1f89d3f0cd49588788997ecba18080 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc1f89d3f0cd49588788997ecba18080" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43027 } } peers { permanent_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34219 } } peers { permanent_uuid: "37dfcdbd7b6f4745a9c25c117f83aa74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35815 } }
I20260926 03:58:52.103919 8424 leader_election.cc:288] T 4b5fcb2756ae4ac5bcfd97ad5ccb464f P cc1f89d3f0cd49588788997ecba18080 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bcf2b7fca8ce4133ac3c49f74265078a (127.0.98.2:34219), 37dfcdbd7b6f4745a9c25c117f83aa74 (127.0.98.3:35815)
I20260926 03:58:52.107863 8485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:52.104068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35815 (local address 127.0.98.1:56131)
0926 03:58:52.104260 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:52.104272 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:52.104351 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:52.104531 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:52.104532 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:52.104534 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:58:52.104701 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:52.104703 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:52.105632 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:52.105633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:52.106968 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:52.106970 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:52.107635 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:52.107636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:52.107698 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:52.107702 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 03:58:52.107759 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 03:58:52.107791 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":93,"thread_start_us":67,"threads_started":1}
I20260926 03:58:52.107903 8487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:52.104269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56131 (local address 127.0.98.3:35815)
0926 03:58:52.104398 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:52.104400 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:52.104408 (+ 8us) server_negotiation.cc:407] Connection header received
0926 03:58:52.104435 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:52.104435 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:52.104446 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:52.104528 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:58:52.104805 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:52.105548 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:52.107051 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:52.107539 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:52.107621 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:52.107633 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:52.107774 (+ 141us) server_negotiation.cc:299] Negotiation successful
0926 03:58:52.107816 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":61,"threads_started":1}
I20260926 03:58:52.108178 8343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b5fcb2756ae4ac5bcfd97ad5ccb464f" candidate_uuid: "cc1f89d3f0cd49588788997ecba18080" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37dfcdbd7b6f4745a9c25c117f83aa74" is_pre_election: true
I20260926 03:58:52.108281 8343 raft_consensus.cc:2454] T 4b5fcb2756ae4ac5bcfd97ad5ccb464f P 37dfcdbd7b6f4745a9c25c117f83aa74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc1f89d3f0cd49588788997ecba18080 in term 0.
I20260926 03:58:52.108349 8486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:52.104068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34219 (local address 127.0.98.1:34935)
0926 03:58:52.104290 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:52.104303 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:52.104370 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:52.104532 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:52.104533 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:52.104535 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:58:52.104650 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:52.104652 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:52.105608 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:52.105610 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:52.107253 (+ 1643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:52.107254 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:52.107945 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:52.107946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:52.108158 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:52.108163 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 03:58:52.108223 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 03:58:52.108265 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":7,"thread_start_us":60,"threads_started":1}
I20260926 03:58:52.108393 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:52.104178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34935 (local address 127.0.98.2:34219)
0926 03:58:52.104228 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:52.104231 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:52.104362 (+ 131us) server_negotiation.cc:407] Connection header received
0926 03:58:52.104431 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:52.104432 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:52.104445 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:52.104524 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:58:52.104733 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:52.105476 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:52.107334 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:52.107845 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:52.107913 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:52.107925 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:52.108240 (+ 315us) server_negotiation.cc:299] Negotiation successful
0926 03:58:52.108283 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 03:58:52.108484 7996 leader_election.cc:302] T 4b5fcb2756ae4ac5bcfd97ad5ccb464f P cc1f89d3f0cd49588788997ecba18080 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37dfcdbd7b6f4745a9c25c117f83aa74, cc1f89d3f0cd49588788997ecba18080; no voters:
I20260926 03:58:52.108611 8424 raft_consensus.cc:2781] T 4b5fcb2756ae4ac5bcfd97ad5ccb464f P cc1f89d3f0cd49588788997ecba18080 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:58:52.108691 8424 raft_consensus.cc:482] T 4b5fcb2756ae4ac5bcfd97ad5ccb464f P cc1f89d3f0cd49588788997ecba18080 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:58:52.108708 8424 raft_consensus.cc:3037] T 4b5fcb2756ae4ac5bcfd97ad5ccb464f P cc1f89d3f0cd49588788997ecba18080 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:58:52.108592 8204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b5fcb2756ae4ac5bcfd97ad5ccb464f" candidate_uuid: "cc1f89d3f0cd49588788997ecba18080" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" is_pre_election: true
I20260926 03:58:52.108759 8204 raft_consensus.cc:2454] T 4b5fcb2756ae4ac5bcfd97ad5ccb464f P bcf2b7fca8ce4133ac3c49f74265078a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc1f89d3f0cd49588788997ecba18080 in term 0.
I20260926 03:58:52.109548 8424 raft_consensus.cc:504] T 4b5fcb2756ae4ac5bcfd97ad5ccb464f P cc1f89d3f0cd49588788997ecba18080 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc1f89d3f0cd49588788997ecba18080" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43027 } } peers { permanent_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34219 } } peers { permanent_uuid: "37dfcdbd7b6f4745a9c25c117f83aa74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35815 } }
I20260926 03:58:52.109695 8424 leader_election.cc:288] T 4b5fcb2756ae4ac5bcfd97ad5ccb464f P cc1f89d3f0cd49588788997ecba18080 [CANDIDATE]: Term 1 election: Requested vote from peers bcf2b7fca8ce4133ac3c49f74265078a (127.0.98.2:34219), 37dfcdbd7b6f4745a9c25c117f83aa74 (127.0.98.3:35815)
I20260926 03:58:52.109835 8343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b5fcb2756ae4ac5bcfd97ad5ccb464f" candidate_uuid: "cc1f89d3f0cd49588788997ecba18080" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37dfcdbd7b6f4745a9c25c117f83aa74"
I20260926 03:58:52.109892 8343 raft_consensus.cc:3037] T 4b5fcb2756ae4ac5bcfd97ad5ccb464f P 37dfcdbd7b6f4745a9c25c117f83aa74 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:58:52.110589 8343 raft_consensus.cc:2454] T 4b5fcb2756ae4ac5bcfd97ad5ccb464f P 37dfcdbd7b6f4745a9c25c117f83aa74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc1f89d3f0cd49588788997ecba18080 in term 1.
I20260926 03:58:52.110683 8204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b5fcb2756ae4ac5bcfd97ad5ccb464f" candidate_uuid: "cc1f89d3f0cd49588788997ecba18080" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bcf2b7fca8ce4133ac3c49f74265078a"
I20260926 03:58:52.110772 8204 raft_consensus.cc:3037] T 4b5fcb2756ae4ac5bcfd97ad5ccb464f P bcf2b7fca8ce4133ac3c49f74265078a [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:58:52.110795 7996 leader_election.cc:302] T 4b5fcb2756ae4ac5bcfd97ad5ccb464f P cc1f89d3f0cd49588788997ecba18080 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37dfcdbd7b6f4745a9c25c117f83aa74, cc1f89d3f0cd49588788997ecba18080; no voters:
I20260926 03:58:52.110912 8424 raft_consensus.cc:2781] T 4b5fcb2756ae4ac5bcfd97ad5ccb464f P cc1f89d3f0cd49588788997ecba18080 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:58:52.111063 8424 raft_consensus.cc:686] T 4b5fcb2756ae4ac5bcfd97ad5ccb464f P cc1f89d3f0cd49588788997ecba18080 [term 1 LEADER]: Becoming Leader. State: Replica: cc1f89d3f0cd49588788997ecba18080, State: Running, Role: LEADER
I20260926 03:58:52.111140 8424 consensus_queue.cc:237] T 4b5fcb2756ae4ac5bcfd97ad5ccb464f P cc1f89d3f0cd49588788997ecba18080 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc1f89d3f0cd49588788997ecba18080" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43027 } } peers { permanent_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34219 } } peers { permanent_uuid: "37dfcdbd7b6f4745a9c25c117f83aa74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35815 } }
I20260926 03:58:52.111588 8204 raft_consensus.cc:2454] T 4b5fcb2756ae4ac5bcfd97ad5ccb464f P bcf2b7fca8ce4133ac3c49f74265078a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc1f89d3f0cd49588788997ecba18080 in term 1.
I20260926 03:58:52.111845 7906 catalog_manager.cc:5495] T 4b5fcb2756ae4ac5bcfd97ad5ccb464f P cc1f89d3f0cd49588788997ecba18080 reported cstate change: term changed from 0 to 1, leader changed from <none> to cc1f89d3f0cd49588788997ecba18080 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "cc1f89d3f0cd49588788997ecba18080" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc1f89d3f0cd49588788997ecba18080" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43027 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34219 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37dfcdbd7b6f4745a9c25c117f83aa74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35815 } health_report { overall_health: UNKNOWN } } }
I20260926 03:58:52.116492 8491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:52.114311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50414 (local address 127.0.98.1:43027)
0926 03:58:52.114427 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:52.114431 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:52.114443 (+ 12us) server_negotiation.cc:407] Connection header received
0926 03:58:52.114471 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:52.114471 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:52.114486 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:52.114554 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:58:52.114654 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:52.115111 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:52.115772 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:52.115948 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:52.116011 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:52.116188 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:58:52.116275 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:58:52.116331 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:52.116370 (+ 39us) server_negotiation.cc:299] Negotiation successful
0926 03:58:52.116384 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":74,"threads_started":1}
I20260926 03:58:52.117333 8343 raft_consensus.cc:1261] T 4b5fcb2756ae4ac5bcfd97ad5ccb464f P 37dfcdbd7b6f4745a9c25c117f83aa74 [term 1 FOLLOWER]: Refusing update from remote peer cc1f89d3f0cd49588788997ecba18080: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:58:52.117547 8204 raft_consensus.cc:1261] T 4b5fcb2756ae4ac5bcfd97ad5ccb464f P bcf2b7fca8ce4133ac3c49f74265078a [term 1 FOLLOWER]: Refusing update from remote peer cc1f89d3f0cd49588788997ecba18080: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:58:52.117592 8488 consensus_queue.cc:1035] T 4b5fcb2756ae4ac5bcfd97ad5ccb464f P cc1f89d3f0cd49588788997ecba18080 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37dfcdbd7b6f4745a9c25c117f83aa74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:58:52.117800 8488 consensus_queue.cc:1035] T 4b5fcb2756ae4ac5bcfd97ad5ccb464f P cc1f89d3f0cd49588788997ecba18080 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:58:52.120869 7906 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:58:52.120965 7906 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:58:52.121519 7906 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bac06df8dabe430ca2575d7abbe118ac: Sending DeleteTablet for 3 replicas of tablet 4b5fcb2756ae4ac5bcfd97ad5ccb464f
I20260926 03:58:52.121799 8324 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b5fcb2756ae4ac5bcfd97ad5ccb464f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:58:52 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43464
I20260926 03:58:52.121841 8044 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b5fcb2756ae4ac5bcfd97ad5ccb464f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:58:52 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50406
I20260926 03:58:52.121920 8440 tablet_replica.cc:330] T 4b5fcb2756ae4ac5bcfd97ad5ccb464f P cc1f89d3f0cd49588788997ecba18080: stopping tablet replica
I20260926 03:58:52.121928 8183 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b5fcb2756ae4ac5bcfd97ad5ccb464f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:58:52 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33410
I20260926 03:58:52.121987 8440 raft_consensus.cc:2229] T 4b5fcb2756ae4ac5bcfd97ad5ccb464f P cc1f89d3f0cd49588788997ecba18080 [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:58:52.122010 8441 tablet_replica.cc:330] T 4b5fcb2756ae4ac5bcfd97ad5ccb464f P 37dfcdbd7b6f4745a9c25c117f83aa74: stopping tablet replica
I20260926 03:58:52.122061 8441 raft_consensus.cc:2229] T 4b5fcb2756ae4ac5bcfd97ad5ccb464f P 37dfcdbd7b6f4745a9c25c117f83aa74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:58:52.122084 8440 raft_consensus.cc:2258] T 4b5fcb2756ae4ac5bcfd97ad5ccb464f P cc1f89d3f0cd49588788997ecba18080 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:58:52.122108 8441 raft_consensus.cc:2258] T 4b5fcb2756ae4ac5bcfd97ad5ccb464f P 37dfcdbd7b6f4745a9c25c117f83aa74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:58:52.122186 8442 tablet_replica.cc:330] T 4b5fcb2756ae4ac5bcfd97ad5ccb464f P bcf2b7fca8ce4133ac3c49f74265078a: stopping tablet replica
I20260926 03:58:52.122242 8442 raft_consensus.cc:2229] T 4b5fcb2756ae4ac5bcfd97ad5ccb464f P bcf2b7fca8ce4133ac3c49f74265078a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:58:52.122279 8442 raft_consensus.cc:2258] T 4b5fcb2756ae4ac5bcfd97ad5ccb464f P bcf2b7fca8ce4133ac3c49f74265078a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:58:52.122316 8440 ts_tablet_manager.cc:1905] T 4b5fcb2756ae4ac5bcfd97ad5ccb464f P cc1f89d3f0cd49588788997ecba18080: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:58:52.122759 8441 ts_tablet_manager.cc:1905] T 4b5fcb2756ae4ac5bcfd97ad5ccb464f P 37dfcdbd7b6f4745a9c25c117f83aa74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:58:52.122762 7906 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33510:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:58:52.122889 7906 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:58:52.123876 8440 ts_tablet_manager.cc:1918] T 4b5fcb2756ae4ac5bcfd97ad5ccb464f P cc1f89d3f0cd49588788997ecba18080: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:58:52.123934 8440 log.cc:1196] T 4b5fcb2756ae4ac5bcfd97ad5ccb464f P cc1f89d3f0cd49588788997ecba18080: Deleting WAL directory at /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/minicluster-data/ts-0/wal/wals/4b5fcb2756ae4ac5bcfd97ad5ccb464f
I20260926 03:58:52.124186 8440 ts_tablet_manager.cc:1939] T 4b5fcb2756ae4ac5bcfd97ad5ccb464f P cc1f89d3f0cd49588788997ecba18080: Deleting consensus metadata
I20260926 03:58:52.124222 8441 ts_tablet_manager.cc:1918] T 4b5fcb2756ae4ac5bcfd97ad5ccb464f P 37dfcdbd7b6f4745a9c25c117f83aa74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:58:52.124269 8441 log.cc:1196] T 4b5fcb2756ae4ac5bcfd97ad5ccb464f P 37dfcdbd7b6f4745a9c25c117f83aa74: Deleting WAL directory at /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/minicluster-data/ts-2/wal/wals/4b5fcb2756ae4ac5bcfd97ad5ccb464f
I20260926 03:58:52.124509 8441 ts_tablet_manager.cc:1939] T 4b5fcb2756ae4ac5bcfd97ad5ccb464f P 37dfcdbd7b6f4745a9c25c117f83aa74: Deleting consensus metadata
I20260926 03:58:52.124887 8324 tablet_service.cc:1467] Processing CreateTablet for tablet cd2ec093f6a64e63ace28831ed9e3a04 (DEFAULT_TABLE table=test-table [id=41e7319eca28424c8dc72d499019f162]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:58:52.124924 8442 ts_tablet_manager.cc:1905] T 4b5fcb2756ae4ac5bcfd97ad5ccb464f P bcf2b7fca8ce4133ac3c49f74265078a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:58:52.124987 8324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd2ec093f6a64e63ace28831ed9e3a04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:58:52.126102 8419 tablet_bootstrap.cc:492] T cd2ec093f6a64e63ace28831ed9e3a04 P 37dfcdbd7b6f4745a9c25c117f83aa74: Bootstrap starting.
I20260926 03:58:52.126302 8442 ts_tablet_manager.cc:1918] T 4b5fcb2756ae4ac5bcfd97ad5ccb464f P bcf2b7fca8ce4133ac3c49f74265078a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:58:52.126348 8442 log.cc:1196] T 4b5fcb2756ae4ac5bcfd97ad5ccb464f P bcf2b7fca8ce4133ac3c49f74265078a: Deleting WAL directory at /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/minicluster-data/ts-1/wal/wals/4b5fcb2756ae4ac5bcfd97ad5ccb464f
I20260926 03:58:52.126477 8419 tablet_bootstrap.cc:654] T cd2ec093f6a64e63ace28831ed9e3a04 P 37dfcdbd7b6f4745a9c25c117f83aa74: Neither blocks nor log segments found. Creating new log.
I20260926 03:58:52.126537 8442 ts_tablet_manager.cc:1939] T 4b5fcb2756ae4ac5bcfd97ad5ccb464f P bcf2b7fca8ce4133ac3c49f74265078a: Deleting consensus metadata
I20260926 03:58:52.126770 7892 catalog_manager.cc:4841] TS 37dfcdbd7b6f4745a9c25c117f83aa74 (127.0.98.3:35815): tablet 4b5fcb2756ae4ac5bcfd97ad5ccb464f (table test-table [id=f4a08b560cab4c47b477933f0fb2646d]) successfully deleted
I20260926 03:58:52.127005 7894 catalog_manager.cc:4841] TS bcf2b7fca8ce4133ac3c49f74265078a (127.0.98.2:34219): tablet 4b5fcb2756ae4ac5bcfd97ad5ccb464f (table test-table [id=f4a08b560cab4c47b477933f0fb2646d]) successfully deleted
I20260926 03:58:52.127056 7894 catalog_manager.cc:4841] TS cc1f89d3f0cd49588788997ecba18080 (127.0.98.1:43027): tablet 4b5fcb2756ae4ac5bcfd97ad5ccb464f (table test-table [id=f4a08b560cab4c47b477933f0fb2646d]) successfully deleted
I20260926 03:58:52.127072 8044 tablet_service.cc:1467] Processing CreateTablet for tablet cd2ec093f6a64e63ace28831ed9e3a04 (DEFAULT_TABLE table=test-table [id=41e7319eca28424c8dc72d499019f162]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:58:52.127164 8044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd2ec093f6a64e63ace28831ed9e3a04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:58:52.127660 8419 tablet_bootstrap.cc:492] T cd2ec093f6a64e63ace28831ed9e3a04 P 37dfcdbd7b6f4745a9c25c117f83aa74: No bootstrap required, opened a new log
I20260926 03:58:52.127748 8419 ts_tablet_manager.cc:1397] T cd2ec093f6a64e63ace28831ed9e3a04 P 37dfcdbd7b6f4745a9c25c117f83aa74: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 03:58:52.127902 8419 raft_consensus.cc:348] T cd2ec093f6a64e63ace28831ed9e3a04 P 37dfcdbd7b6f4745a9c25c117f83aa74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc1f89d3f0cd49588788997ecba18080" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43027 } } peers { permanent_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34219 } } peers { permanent_uuid: "37dfcdbd7b6f4745a9c25c117f83aa74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35815 } }
I20260926 03:58:52.127967 8419 raft_consensus.cc:374] T cd2ec093f6a64e63ace28831ed9e3a04 P 37dfcdbd7b6f4745a9c25c117f83aa74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:58:52.127988 8419 raft_consensus.cc:729] T cd2ec093f6a64e63ace28831ed9e3a04 P 37dfcdbd7b6f4745a9c25c117f83aa74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37dfcdbd7b6f4745a9c25c117f83aa74, State: Initialized, Role: FOLLOWER
I20260926 03:58:52.128026 8419 consensus_queue.cc:260] T cd2ec093f6a64e63ace28831ed9e3a04 P 37dfcdbd7b6f4745a9c25c117f83aa74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc1f89d3f0cd49588788997ecba18080" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43027 } } peers { permanent_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34219 } } peers { permanent_uuid: "37dfcdbd7b6f4745a9c25c117f83aa74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35815 } }
I20260926 03:58:52.128120 8419 ts_tablet_manager.cc:1428] T cd2ec093f6a64e63ace28831ed9e3a04 P 37dfcdbd7b6f4745a9c25c117f83aa74: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:52.128288 8418 tablet_bootstrap.cc:492] T cd2ec093f6a64e63ace28831ed9e3a04 P cc1f89d3f0cd49588788997ecba18080: Bootstrap starting.
I20260926 03:58:52.128712 8418 tablet_bootstrap.cc:654] T cd2ec093f6a64e63ace28831ed9e3a04 P cc1f89d3f0cd49588788997ecba18080: Neither blocks nor log segments found. Creating new log.
I20260926 03:58:52.129130 8183 tablet_service.cc:1467] Processing CreateTablet for tablet cd2ec093f6a64e63ace28831ed9e3a04 (DEFAULT_TABLE table=test-table [id=41e7319eca28424c8dc72d499019f162]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:58:52.129231 8183 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd2ec093f6a64e63ace28831ed9e3a04. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:58:52.130002 8420 tablet_bootstrap.cc:492] T cd2ec093f6a64e63ace28831ed9e3a04 P bcf2b7fca8ce4133ac3c49f74265078a: Bootstrap starting.
I20260926 03:58:52.130247 8418 tablet_bootstrap.cc:492] T cd2ec093f6a64e63ace28831ed9e3a04 P cc1f89d3f0cd49588788997ecba18080: No bootstrap required, opened a new log
I20260926 03:58:52.130287 8420 tablet_bootstrap.cc:654] T cd2ec093f6a64e63ace28831ed9e3a04 P bcf2b7fca8ce4133ac3c49f74265078a: Neither blocks nor log segments found. Creating new log.
I20260926 03:58:52.130306 8418 ts_tablet_manager.cc:1397] T cd2ec093f6a64e63ace28831ed9e3a04 P cc1f89d3f0cd49588788997ecba18080: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 03:58:52.130470 8418 raft_consensus.cc:348] T cd2ec093f6a64e63ace28831ed9e3a04 P cc1f89d3f0cd49588788997ecba18080 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc1f89d3f0cd49588788997ecba18080" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43027 } } peers { permanent_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34219 } } peers { permanent_uuid: "37dfcdbd7b6f4745a9c25c117f83aa74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35815 } }
I20260926 03:58:52.130535 8418 raft_consensus.cc:374] T cd2ec093f6a64e63ace28831ed9e3a04 P cc1f89d3f0cd49588788997ecba18080 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:58:52.130550 8418 raft_consensus.cc:729] T cd2ec093f6a64e63ace28831ed9e3a04 P cc1f89d3f0cd49588788997ecba18080 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc1f89d3f0cd49588788997ecba18080, State: Initialized, Role: FOLLOWER
I20260926 03:58:52.130584 8418 consensus_queue.cc:260] T cd2ec093f6a64e63ace28831ed9e3a04 P cc1f89d3f0cd49588788997ecba18080 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc1f89d3f0cd49588788997ecba18080" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43027 } } peers { permanent_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34219 } } peers { permanent_uuid: "37dfcdbd7b6f4745a9c25c117f83aa74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35815 } }
I20260926 03:58:52.130685 8418 ts_tablet_manager.cc:1428] T cd2ec093f6a64e63ace28831ed9e3a04 P cc1f89d3f0cd49588788997ecba18080: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:52.131057 8420 tablet_bootstrap.cc:492] T cd2ec093f6a64e63ace28831ed9e3a04 P bcf2b7fca8ce4133ac3c49f74265078a: No bootstrap required, opened a new log
I20260926 03:58:52.131112 8420 ts_tablet_manager.cc:1397] T cd2ec093f6a64e63ace28831ed9e3a04 P bcf2b7fca8ce4133ac3c49f74265078a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 03:58:52.131204 8420 raft_consensus.cc:348] T cd2ec093f6a64e63ace28831ed9e3a04 P bcf2b7fca8ce4133ac3c49f74265078a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc1f89d3f0cd49588788997ecba18080" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43027 } } peers { permanent_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34219 } } peers { permanent_uuid: "37dfcdbd7b6f4745a9c25c117f83aa74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35815 } }
I20260926 03:58:52.131263 8420 raft_consensus.cc:374] T cd2ec093f6a64e63ace28831ed9e3a04 P bcf2b7fca8ce4133ac3c49f74265078a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:58:52.131282 8420 raft_consensus.cc:729] T cd2ec093f6a64e63ace28831ed9e3a04 P bcf2b7fca8ce4133ac3c49f74265078a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf2b7fca8ce4133ac3c49f74265078a, State: Initialized, Role: FOLLOWER
I20260926 03:58:52.131326 8420 consensus_queue.cc:260] T cd2ec093f6a64e63ace28831ed9e3a04 P bcf2b7fca8ce4133ac3c49f74265078a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc1f89d3f0cd49588788997ecba18080" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43027 } } peers { permanent_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34219 } } peers { permanent_uuid: "37dfcdbd7b6f4745a9c25c117f83aa74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35815 } }
I20260926 03:58:52.131409 8420 ts_tablet_manager.cc:1428] T cd2ec093f6a64e63ace28831ed9e3a04 P bcf2b7fca8ce4133ac3c49f74265078a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:52.358392 8462 raft_consensus.cc:482] T cd2ec093f6a64e63ace28831ed9e3a04 P bcf2b7fca8ce4133ac3c49f74265078a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:58:52.358491 8462 raft_consensus.cc:504] T cd2ec093f6a64e63ace28831ed9e3a04 P bcf2b7fca8ce4133ac3c49f74265078a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc1f89d3f0cd49588788997ecba18080" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43027 } } peers { permanent_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34219 } } peers { permanent_uuid: "37dfcdbd7b6f4745a9c25c117f83aa74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35815 } }
I20260926 03:58:52.358678 8462 leader_election.cc:288] T cd2ec093f6a64e63ace28831ed9e3a04 P bcf2b7fca8ce4133ac3c49f74265078a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc1f89d3f0cd49588788997ecba18080 (127.0.98.1:43027), 37dfcdbd7b6f4745a9c25c117f83aa74 (127.0.98.3:35815)
I20260926 03:58:52.358891 8064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2ec093f6a64e63ace28831ed9e3a04" candidate_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc1f89d3f0cd49588788997ecba18080" is_pre_election: true
I20260926 03:58:52.358901 8343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2ec093f6a64e63ace28831ed9e3a04" candidate_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37dfcdbd7b6f4745a9c25c117f83aa74" is_pre_election: true
I20260926 03:58:52.358985 8064 raft_consensus.cc:2454] T cd2ec093f6a64e63ace28831ed9e3a04 P cc1f89d3f0cd49588788997ecba18080 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf2b7fca8ce4133ac3c49f74265078a in term 0.
I20260926 03:58:52.359015 8343 raft_consensus.cc:2454] T cd2ec093f6a64e63ace28831ed9e3a04 P 37dfcdbd7b6f4745a9c25c117f83aa74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf2b7fca8ce4133ac3c49f74265078a in term 0.
I20260926 03:58:52.359220 8136 leader_election.cc:302] T cd2ec093f6a64e63ace28831ed9e3a04 P bcf2b7fca8ce4133ac3c49f74265078a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37dfcdbd7b6f4745a9c25c117f83aa74, bcf2b7fca8ce4133ac3c49f74265078a; no voters:
I20260926 03:58:52.359359 8462 raft_consensus.cc:2781] T cd2ec093f6a64e63ace28831ed9e3a04 P bcf2b7fca8ce4133ac3c49f74265078a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:58:52.359409 8462 raft_consensus.cc:482] T cd2ec093f6a64e63ace28831ed9e3a04 P bcf2b7fca8ce4133ac3c49f74265078a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:58:52.359428 8462 raft_consensus.cc:3037] T cd2ec093f6a64e63ace28831ed9e3a04 P bcf2b7fca8ce4133ac3c49f74265078a [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:58:52.360144 8462 raft_consensus.cc:504] T cd2ec093f6a64e63ace28831ed9e3a04 P bcf2b7fca8ce4133ac3c49f74265078a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc1f89d3f0cd49588788997ecba18080" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43027 } } peers { permanent_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34219 } } peers { permanent_uuid: "37dfcdbd7b6f4745a9c25c117f83aa74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35815 } }
I20260926 03:58:52.360280 8462 leader_election.cc:288] T cd2ec093f6a64e63ace28831ed9e3a04 P bcf2b7fca8ce4133ac3c49f74265078a [CANDIDATE]: Term 1 election: Requested vote from peers cc1f89d3f0cd49588788997ecba18080 (127.0.98.1:43027), 37dfcdbd7b6f4745a9c25c117f83aa74 (127.0.98.3:35815)
I20260926 03:58:52.360399 8064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2ec093f6a64e63ace28831ed9e3a04" candidate_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc1f89d3f0cd49588788997ecba18080"
I20260926 03:58:52.360412 8343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd2ec093f6a64e63ace28831ed9e3a04" candidate_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37dfcdbd7b6f4745a9c25c117f83aa74"
I20260926 03:58:52.360476 8064 raft_consensus.cc:3037] T cd2ec093f6a64e63ace28831ed9e3a04 P cc1f89d3f0cd49588788997ecba18080 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:58:52.360476 8343 raft_consensus.cc:3037] T cd2ec093f6a64e63ace28831ed9e3a04 P 37dfcdbd7b6f4745a9c25c117f83aa74 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:58:52.361140 8064 raft_consensus.cc:2454] T cd2ec093f6a64e63ace28831ed9e3a04 P cc1f89d3f0cd49588788997ecba18080 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf2b7fca8ce4133ac3c49f74265078a in term 1.
I20260926 03:58:52.361171 8343 raft_consensus.cc:2454] T cd2ec093f6a64e63ace28831ed9e3a04 P 37dfcdbd7b6f4745a9c25c117f83aa74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf2b7fca8ce4133ac3c49f74265078a in term 1.
I20260926 03:58:52.361336 8138 leader_election.cc:302] T cd2ec093f6a64e63ace28831ed9e3a04 P bcf2b7fca8ce4133ac3c49f74265078a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bcf2b7fca8ce4133ac3c49f74265078a, cc1f89d3f0cd49588788997ecba18080; no voters:
I20260926 03:58:52.361439 8462 raft_consensus.cc:2781] T cd2ec093f6a64e63ace28831ed9e3a04 P bcf2b7fca8ce4133ac3c49f74265078a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:58:52.361495 8462 raft_consensus.cc:686] T cd2ec093f6a64e63ace28831ed9e3a04 P bcf2b7fca8ce4133ac3c49f74265078a [term 1 LEADER]: Becoming Leader. State: Replica: bcf2b7fca8ce4133ac3c49f74265078a, State: Running, Role: LEADER
I20260926 03:58:52.361548 8462 consensus_queue.cc:237] T cd2ec093f6a64e63ace28831ed9e3a04 P bcf2b7fca8ce4133ac3c49f74265078a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc1f89d3f0cd49588788997ecba18080" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43027 } } peers { permanent_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34219 } } peers { permanent_uuid: "37dfcdbd7b6f4745a9c25c117f83aa74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35815 } }
I20260926 03:58:52.362175 7906 catalog_manager.cc:5495] T cd2ec093f6a64e63ace28831ed9e3a04 P bcf2b7fca8ce4133ac3c49f74265078a reported cstate change: term changed from 0 to 1, leader changed from <none> to bcf2b7fca8ce4133ac3c49f74265078a (127.0.98.2). New cstate: current_term: 1 leader_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cc1f89d3f0cd49588788997ecba18080" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf2b7fca8ce4133ac3c49f74265078a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34219 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "37dfcdbd7b6f4745a9c25c117f83aa74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35815 } health_report { overall_health: UNKNOWN } } }
I20260926 03:58:52.413805 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:52.411491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35274 (local address 127.0.98.2:34219)
0926 03:58:52.411553 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:52.411556 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:52.411574 (+ 18us) server_negotiation.cc:407] Connection header received
0926 03:58:52.411591 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:52.411592 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:52.411604 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:52.411662 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:58:52.411807 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:52.412285 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:52.413097 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:52.413306 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:52.413389 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:52.413529 (+ 140us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:58:52.413562 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:58:52.413639 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:52.413699 (+ 60us) server_negotiation.cc:299] Negotiation successful
0926 03:58:52.413718 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 03:58:52.415907 8343 raft_consensus.cc:1261] T cd2ec093f6a64e63ace28831ed9e3a04 P 37dfcdbd7b6f4745a9c25c117f83aa74 [term 1 FOLLOWER]: Refusing update from remote peer bcf2b7fca8ce4133ac3c49f74265078a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:58:52.415942 8064 raft_consensus.cc:1261] T cd2ec093f6a64e63ace28831ed9e3a04 P cc1f89d3f0cd49588788997ecba18080 [term 1 FOLLOWER]: Refusing update from remote peer bcf2b7fca8ce4133ac3c49f74265078a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:58:52.416149 8466 consensus_queue.cc:1035] T cd2ec093f6a64e63ace28831ed9e3a04 P bcf2b7fca8ce4133ac3c49f74265078a [LEADER]: Connected to new peer: Peer: permanent_uuid: "37dfcdbd7b6f4745a9c25c117f83aa74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:58:52.416263 8462 consensus_queue.cc:1035] T cd2ec093f6a64e63ace28831ed9e3a04 P bcf2b7fca8ce4133ac3c49f74265078a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc1f89d3f0cd49588788997ecba18080" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:58:52.426091 7907 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:58:52.426168 7907 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:58:52.426770 7907 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bac06df8dabe430ca2575d7abbe118ac: Sending DeleteTablet for 3 replicas of tablet cd2ec093f6a64e63ace28831ed9e3a04
I20260926 03:58:52.426987 8044 tablet_service.cc:1514] Processing DeleteTablet for tablet cd2ec093f6a64e63ace28831ed9e3a04 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:58:52 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50406
I20260926 03:58:52.427083 8440 tablet_replica.cc:330] T cd2ec093f6a64e63ace28831ed9e3a04 P cc1f89d3f0cd49588788997ecba18080: stopping tablet replica
I20260926 03:58:52.427054 8324 tablet_service.cc:1514] Processing DeleteTablet for tablet cd2ec093f6a64e63ace28831ed9e3a04 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:58:52 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43464
I20260926 03:58:52.427135 8440 raft_consensus.cc:2229] T cd2ec093f6a64e63ace28831ed9e3a04 P cc1f89d3f0cd49588788997ecba18080 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:58:52.427168 8440 raft_consensus.cc:2258] T cd2ec093f6a64e63ace28831ed9e3a04 P cc1f89d3f0cd49588788997ecba18080 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:58:52.427201 8441 tablet_replica.cc:330] T cd2ec093f6a64e63ace28831ed9e3a04 P 37dfcdbd7b6f4745a9c25c117f83aa74: stopping tablet replica
I20260926 03:58:52.427260 8441 raft_consensus.cc:2229] T cd2ec093f6a64e63ace28831ed9e3a04 P 37dfcdbd7b6f4745a9c25c117f83aa74 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:58:52.427299 8441 raft_consensus.cc:2258] T cd2ec093f6a64e63ace28831ed9e3a04 P 37dfcdbd7b6f4745a9c25c117f83aa74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:58:52.427575 8440 ts_tablet_manager.cc:1905] T cd2ec093f6a64e63ace28831ed9e3a04 P cc1f89d3f0cd49588788997ecba18080: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:58:52.427631 8441 ts_tablet_manager.cc:1905] T cd2ec093f6a64e63ace28831ed9e3a04 P 37dfcdbd7b6f4745a9c25c117f83aa74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:58:52.428633 8183 tablet_service.cc:1514] Processing DeleteTablet for tablet cd2ec093f6a64e63ace28831ed9e3a04 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:58:52 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33410
I20260926 03:58:52.428771 8442 tablet_replica.cc:330] T cd2ec093f6a64e63ace28831ed9e3a04 P bcf2b7fca8ce4133ac3c49f74265078a: stopping tablet replica
I20260926 03:58:52.428828 8442 raft_consensus.cc:2229] T cd2ec093f6a64e63ace28831ed9e3a04 P bcf2b7fca8ce4133ac3c49f74265078a [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:58:52.428894 8442 raft_consensus.cc:2258] T cd2ec093f6a64e63ace28831ed9e3a04 P bcf2b7fca8ce4133ac3c49f74265078a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:58:52.428925 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 7979
I20260926 03:58:52.429057 8441 ts_tablet_manager.cc:1918] T cd2ec093f6a64e63ace28831ed9e3a04 P 37dfcdbd7b6f4745a9c25c117f83aa74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 03:58:52.429116 8441 log.cc:1196] T cd2ec093f6a64e63ace28831ed9e3a04 P 37dfcdbd7b6f4745a9c25c117f83aa74: Deleting WAL directory at /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/minicluster-data/ts-2/wal/wals/cd2ec093f6a64e63ace28831ed9e3a04
I20260926 03:58:52.429301 8441 ts_tablet_manager.cc:1939] T cd2ec093f6a64e63ace28831ed9e3a04 P 37dfcdbd7b6f4745a9c25c117f83aa74: Deleting consensus metadata
I20260926 03:58:52.430392 7892 catalog_manager.cc:4841] TS 37dfcdbd7b6f4745a9c25c117f83aa74 (127.0.98.3:35815): tablet cd2ec093f6a64e63ace28831ed9e3a04 (table test-table [id=41e7319eca28424c8dc72d499019f162]) successfully deleted
I20260926 03:58:52.430532 8442 ts_tablet_manager.cc:1905] T cd2ec093f6a64e63ace28831ed9e3a04 P bcf2b7fca8ce4133ac3c49f74265078a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:58:52.432261 8442 ts_tablet_manager.cc:1918] T cd2ec093f6a64e63ace28831ed9e3a04 P bcf2b7fca8ce4133ac3c49f74265078a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 03:58:52.432325 8442 log.cc:1196] T cd2ec093f6a64e63ace28831ed9e3a04 P bcf2b7fca8ce4133ac3c49f74265078a: Deleting WAL directory at /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790395095474643-392-0/minicluster-data/ts-1/wal/wals/cd2ec093f6a64e63ace28831ed9e3a04
I20260926 03:58:52.432520 8442 ts_tablet_manager.cc:1939] T cd2ec093f6a64e63ace28831ed9e3a04 P bcf2b7fca8ce4133ac3c49f74265078a: Deleting consensus metadata
I20260926 03:58:52.433044 7894 catalog_manager.cc:4841] TS bcf2b7fca8ce4133ac3c49f74265078a (127.0.98.2:34219): tablet cd2ec093f6a64e63ace28831ed9e3a04 (table test-table [id=41e7319eca28424c8dc72d499019f162]) successfully deleted
W20260926 03:58:52.435786 7894 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43027 (error 108)
I20260926 03:58:52.436230 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 8119
I20260926 03:58:52.436342 8503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:52.436174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43027 (local address 127.0.0.1:58724)
0926 03:58:52.436253 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:52.436298 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62,"thread_start_us":43,"threads_started":1}
W20260926 03:58:52.437031 7894 catalog_manager.cc:4584] TS cc1f89d3f0cd49588788997ecba18080 (127.0.98.1:43027): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cd2ec093f6a64e63ace28831ed9e3a04: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43027: connect: Connection refused (error 111)
I20260926 03:58:52.442884 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 8259
I20260926 03:58:52.447610 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 7875
2026-09-26T03:58:52Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2905 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790395095474643-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:58:52 dist-test-slave-nhpw krb5kdc[8506](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:58:52 dist-test-slave-nhpw krb5kdc[8506](info): set up 2 sockets
Sep 26 03:58:52 dist-test-slave-nhpw krb5kdc[8506](info): commencing operation
krb5kdc: starting...
W20260926 03:58:53.485060 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790395095474643-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790395095474643-392-0/krb5kdc/test-user.keytab.
Sep 26 03:58:53 dist-test-slave-nhpw krb5kdc[8506](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395133, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:58:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:58:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 03:58:53.599282 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790395095474643-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790395095474643-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790395095474643-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40033
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40033
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:58:53.680904 8519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:58:53.681144 8519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:58:53.681190 8519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:58:53.682310 8519 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:58:53.682384 8519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:58:53.682407 8519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:58:53.682436 8519 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:58:53.682467 8519 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:58:53.683951 8519 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790395095474643-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790395095474643-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40033
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8519
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790395095474643-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:58:53.684230 8519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:58:53.684453 8519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:58:53.689282 8527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:53.689411 8524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:53.689448 8519 server_base.cc:1034] running on GCE node
W20260926 03:58:53.689561 8525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:53.689822 8519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:58:53.690083 8519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:58:53.691228 8519 hybrid_clock.cc:648] HybridClock initialized: now 1790395133691211 us; error 34 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-26T03:58:53Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1248 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:58:53 dist-test-slave-nhpw krb5kdc[8532](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:58:53 dist-test-slave-nhpw krb5kdc[8532](info): set up 2 sockets
Sep 26 03:58:53 dist-test-slave-nhpw krb5kdc[8532](info): commencing operation
krb5kdc: starting...
W20260926 03:58:54.745280 392 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/krb5kdc/test-user.keytab.
Sep 26 03:58:54 dist-test-slave-nhpw krb5kdc[8532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395134, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:58:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:58:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 03:58:54.863569 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45737
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45737
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:58:54.945593 8545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:58:54.945804 8545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:58:54.945827 8545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:58:54.947016 8545 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:58:54.947062 8545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:58:54.947075 8545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:58:54.947088 8545 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:58:54.947099 8545 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:58:54.948689 8545 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45737
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8545
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:58:54.948938 8545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:58:54.949167 8545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:58:54.954027 8553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:54.954032 8551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:54.954205 8545 server_base.cc:1034] running on GCE node
W20260926 03:58:54.954034 8550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:54.954481 8545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:58:54.954778 8545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:58:54.955899 8545 hybrid_clock.cc:648] HybridClock initialized: now 1790395134955900 us; error 49 us; skew 500 ppm
Sep 26 03:58:54 dist-test-slave-nhpw krb5kdc[8532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395134, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:58:54.958395 8545 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 03:58:54.958827 8545 webserver.cc:460] Webserver started at http://127.0.98.62:45099/ using document root <none> and password file <none>
I20260926 03:58:54.959064 8545 fs_manager.cc:359] Metadata directory not provided
I20260926 03:58:54.959131 8545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:58:54.959270 8545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:58:54.960305 8545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/master-0/data/instance:
uuid: "4a97d90c240d4a3cae6ca86026d874cb"
format_stamp: "Formatted at 2026-09-26 03:58:54 on dist-test-slave-nhpw"
server_key: "4acc2b7d3b16856bd1f77ddc983d7907"
server_key_iv: "7ba2ef9371d3e6d4aa94ad80d2e2830d"
server_key_version: "encryptionkey@0"
I20260926 03:58:54.960732 8545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/master-0/wal/instance:
uuid: "4a97d90c240d4a3cae6ca86026d874cb"
format_stamp: "Formatted at 2026-09-26 03:58:54 on dist-test-slave-nhpw"
server_key: "4acc2b7d3b16856bd1f77ddc983d7907"
server_key_iv: "7ba2ef9371d3e6d4aa94ad80d2e2830d"
server_key_version: "encryptionkey@0"
I20260926 03:58:54.962101 8545 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 03:58:54.962885 8560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:54.963084 8545 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 03:58:54.963207 8545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/master-0/wal
uuid: "4a97d90c240d4a3cae6ca86026d874cb"
format_stamp: "Formatted at 2026-09-26 03:58:54 on dist-test-slave-nhpw"
server_key: "4acc2b7d3b16856bd1f77ddc983d7907"
server_key_iv: "7ba2ef9371d3e6d4aa94ad80d2e2830d"
server_key_version: "encryptionkey@0"
I20260926 03:58:54.963289 8545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:58:54.981664 8545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:58:54.983899 8545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:58:54.984083 8545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:58:54.987850 8545 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45737
I20260926 03:58:54.987884 8612 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45737 every 8 connection(s)
I20260926 03:58:54.988404 8545 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
I20260926 03:58:54.989086 8613 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:58:54.991333 8613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a97d90c240d4a3cae6ca86026d874cb: Bootstrap starting.
I20260926 03:58:54.991938 8613 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a97d90c240d4a3cae6ca86026d874cb: Neither blocks nor log segments found. Creating new log.
I20260926 03:58:54.992223 8613 log.cc:824] T 00000000000000000000000000000000 P 4a97d90c240d4a3cae6ca86026d874cb: Log is configured to *not* fsync() on all Append() calls
I20260926 03:58:54.992923 8613 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a97d90c240d4a3cae6ca86026d874cb: No bootstrap required, opened a new log
I20260926 03:58:54.994412 8613 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a97d90c240d4a3cae6ca86026d874cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a97d90c240d4a3cae6ca86026d874cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45737 } }
I20260926 03:58:54.994542 8613 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a97d90c240d4a3cae6ca86026d874cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:58:54.994594 8613 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a97d90c240d4a3cae6ca86026d874cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a97d90c240d4a3cae6ca86026d874cb, State: Initialized, Role: FOLLOWER
I20260926 03:58:54.994740 8613 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a97d90c240d4a3cae6ca86026d874cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a97d90c240d4a3cae6ca86026d874cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45737 } }
I20260926 03:58:54.994799 8613 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a97d90c240d4a3cae6ca86026d874cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:58:54.994820 8613 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a97d90c240d4a3cae6ca86026d874cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:58:54.994884 8613 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a97d90c240d4a3cae6ca86026d874cb [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:58:54.995724 8613 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a97d90c240d4a3cae6ca86026d874cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a97d90c240d4a3cae6ca86026d874cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45737 } }
I20260926 03:58:54.995831 8613 leader_election.cc:302] T 00000000000000000000000000000000 P 4a97d90c240d4a3cae6ca86026d874cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a97d90c240d4a3cae6ca86026d874cb; no voters:
I20260926 03:58:54.996019 8613 leader_election.cc:288] T 00000000000000000000000000000000 P 4a97d90c240d4a3cae6ca86026d874cb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:58:54.996078 8616 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a97d90c240d4a3cae6ca86026d874cb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:58:54.996251 8616 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a97d90c240d4a3cae6ca86026d874cb [term 1 LEADER]: Becoming Leader. State: Replica: 4a97d90c240d4a3cae6ca86026d874cb, State: Running, Role: LEADER
I20260926 03:58:54.996276 8613 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a97d90c240d4a3cae6ca86026d874cb [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:58:54.996335 8616 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a97d90c240d4a3cae6ca86026d874cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a97d90c240d4a3cae6ca86026d874cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45737 } }
I20260926 03:58:54.996711 8617 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a97d90c240d4a3cae6ca86026d874cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a97d90c240d4a3cae6ca86026d874cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a97d90c240d4a3cae6ca86026d874cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45737 } } }
I20260926 03:58:54.996719 8618 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a97d90c240d4a3cae6ca86026d874cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a97d90c240d4a3cae6ca86026d874cb. Latest consensus state: current_term: 1 leader_uuid: "4a97d90c240d4a3cae6ca86026d874cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a97d90c240d4a3cae6ca86026d874cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45737 } } }
I20260926 03:58:54.996835 8617 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a97d90c240d4a3cae6ca86026d874cb [sys.catalog]: This master's current role is: LEADER
I20260926 03:58:54.996876 8618 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a97d90c240d4a3cae6ca86026d874cb [sys.catalog]: This master's current role is: LEADER
I20260926 03:58:54.997099 8625 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:58:54.997640 8625 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:58:54.998543 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 8545
I20260926 03:58:54.998639 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/master-0/wal/instance
I20260926 03:58:54.998838 392 external_mini_cluster.cc:1428] Setting key 60e60157113caf41fbdd57f6b217532d
I20260926 03:58:54.999188 8625 catalog_manager.cc:1347] Generated new cluster ID: a1eba885c4ab4479af34cb12f1b6cb61
I20260926 03:58:55.000604 8625 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 03:58:55 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395134, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 03:58:55.011051 8636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:54.999302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44964 (local address 127.0.98.62:45737)
0926 03:58:54.999921 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:54.999931 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:54.999964 (+ 33us) server_negotiation.cc:407] Connection header received
0926 03:58:55.000123 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:55.000130 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:55.000192 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:55.000329 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:55.001200 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:55.002571 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:55.003672 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:55.004283 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:55.004379 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:55.007306 (+ 2927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:55.007311 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:55.007311 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:55.007324 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:55.009383 (+ 2059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:55.009772 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:55.009775 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:55.009775 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:55.009885 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:55.010087 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:55.010088 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:55.010089 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:55.010395 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:55.010554 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:55.010649 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 03:58:55.010809 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":94,"threads_started":1}
I20260926 03:58:55.011152 8625 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:58:55.011832 8625 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:58:55.018181 8625 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a97d90c240d4a3cae6ca86026d874cb: Generated new TSK 0
I20260926 03:58:55.018342 8625 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 03:58:55.061066 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45737
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:58:55.146613 8639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:58:55.146817 8639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:58:55.146840 8639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:58:55.147922 8639 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:58:55.147956 8639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:58:55.147982 8639 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 03:58:55.149441 8639 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45737
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8639
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:58:55.149617 8639 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:58:55.149796 8639 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:58:55.154826 8645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:55.154906 8639 server_base.cc:1034] running on GCE node
W20260926 03:58:55.154821 8647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:55.154803 8644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:55.155247 8639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:58:55.155485 8639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:58:55.156623 8639 hybrid_clock.cc:648] HybridClock initialized: now 1790395135156611 us; error 27 us; skew 500 ppm
Sep 26 03:58:55 dist-test-slave-nhpw krb5kdc[8532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395135, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:58:55.162595 8639 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 03:58:55.163010 8639 webserver.cc:460] Webserver started at http://127.0.98.1:46107/ using document root <none> and password file <none>
I20260926 03:58:55.163230 8639 fs_manager.cc:359] Metadata directory not provided
I20260926 03:58:55.163273 8639 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:58:55.163411 8639 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:58:55.164438 8639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-0/data/instance:
uuid: "93d0ae3078024164a30f8ce74197d328"
format_stamp: "Formatted at 2026-09-26 03:58:55 on dist-test-slave-nhpw"
server_key: "d4f47c193b28b791479950d8784272b4"
server_key_iv: "3ca7372c393a9d4efb57fdfe8b31f41f"
server_key_version: "encryptionkey@0"
I20260926 03:58:55.164801 8639 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-0/wal/instance:
uuid: "93d0ae3078024164a30f8ce74197d328"
format_stamp: "Formatted at 2026-09-26 03:58:55 on dist-test-slave-nhpw"
server_key: "d4f47c193b28b791479950d8784272b4"
server_key_iv: "3ca7372c393a9d4efb57fdfe8b31f41f"
server_key_version: "encryptionkey@0"
I20260926 03:58:55.166126 8639 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 03:58:55.166824 8654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:55.167021 8639 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 03:58:55.167116 8639 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-0/wal
uuid: "93d0ae3078024164a30f8ce74197d328"
format_stamp: "Formatted at 2026-09-26 03:58:55 on dist-test-slave-nhpw"
server_key: "d4f47c193b28b791479950d8784272b4"
server_key_iv: "3ca7372c393a9d4efb57fdfe8b31f41f"
server_key_version: "encryptionkey@0"
I20260926 03:58:55.167188 8639 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:58:55.173484 8639 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:58:55.175639 8639 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:58:55.175808 8639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:58:55.176039 8639 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:58:55.176363 8639 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:58:55.176415 8639 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:55.176456 8639 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:58:55.176487 8639 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:55.182358 8639 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46801
I20260926 03:58:55.182407 8767 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46801 every 8 connection(s)
I20260926 03:58:55.182699 8639 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
I20260926 03:58:55.185734 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 8639
I20260926 03:58:55.185842 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-0/wal/instance
I20260926 03:58:55.186013 392 external_mini_cluster.cc:1428] Setting key fede563311029dbb6db37af25268589e
Sep 26 03:58:55 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395135, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 03:58:55.193625 8636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:55.183541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54179 (local address 127.0.98.62:45737)
0926 03:58:55.183588 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:55.183590 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:55.183812 (+ 222us) server_negotiation.cc:407] Connection header received
0926 03:58:55.184865 (+ 1053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:55.184866 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:55.184879 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:55.184932 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:55.185609 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:55.186287 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:55.187835 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:55.188176 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:55.188263 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:55.190463 (+ 2200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:55.190465 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:55.190465 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:55.190472 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:55.192218 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:55.192624 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:55.192626 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:55.192626 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:55.192711 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:55.192937 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:55.192937 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:55.192938 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:55.193092 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:55.193167 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:55.193409 (+ 242us) server_negotiation.cc:299] Negotiation successful
0926 03:58:55.193500 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 03:58:55.193643 8770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:55.183522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45737 (local address 127.0.98.1:54179)
0926 03:58:55.183694 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:55.183729 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:55.184753 (+ 1024us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:55.185002 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:55.185007 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:55.185286 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:55.185486 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:55.185491 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:55.186382 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:55.186383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:55.187720 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:55.187724 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:55.188275 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:55.188277 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:55.188381 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:55.188993 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:55.189002 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:55.190359 (+ 1357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:58:55.192304 (+ 1945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:55.192310 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:55.192312 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:55.192544 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:55.192778 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:55.192778 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:55.192778 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:55.192873 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:55.193153 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:58:55.193157 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:58:55.193278 (+ 121us) client_negotiation.cc:769] Sending connection context
0926 03:58:55.193361 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 03:58:55.193482 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":80,"threads_started":1}
I20260926 03:58:55.194190 8768 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45737
I20260926 03:58:55.194291 8768 heartbeater.cc:464] Registering TS with master...
I20260926 03:58:55.194480 8768 heartbeater.cc:511] Master 127.0.98.62:45737 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 03:58:55.195003 8577 ts_manager.cc:194] Registered new tserver with Master: 93d0ae3078024164a30f8ce74197d328 (127.0.98.1:46801)
I20260926 03:58:55.195628 8577 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54179
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 03:58:55.225124 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45737
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:58:55.305430 8773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:58:55.305662 8773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:58:55.305699 8773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:58:55.306878 8773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:58:55.306943 8773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:58:55.306988 8773 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 03:58:55.308503 8773 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45737
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8773
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:58:55.308770 8773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:58:55.308988 8773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:58:55.313862 8778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:55.313870 8781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:55.313882 8779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:55.314050 8773 server_base.cc:1034] running on GCE node
I20260926 03:58:55.314322 8773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:58:55.314584 8773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:58:55.315742 8773 hybrid_clock.cc:648] HybridClock initialized: now 1790395135315723 us; error 32 us; skew 500 ppm
Sep 26 03:58:55 dist-test-slave-nhpw krb5kdc[8532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395135, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:58:55.318334 8773 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 03:58:55.318740 8773 webserver.cc:460] Webserver started at http://127.0.98.2:33947/ using document root <none> and password file <none>
I20260926 03:58:55.318962 8773 fs_manager.cc:359] Metadata directory not provided
I20260926 03:58:55.319051 8773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:58:55.319188 8773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:58:55.320183 8773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-1/data/instance:
uuid: "e91ee3ee2d844e298dd7f3a7e586502c"
format_stamp: "Formatted at 2026-09-26 03:58:55 on dist-test-slave-nhpw"
server_key: "5888e79a7fb95e51562995c74bd07469"
server_key_iv: "421c6e200eae0c937d8fb694fb685001"
server_key_version: "encryptionkey@0"
I20260926 03:58:55.320528 8773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e91ee3ee2d844e298dd7f3a7e586502c"
format_stamp: "Formatted at 2026-09-26 03:58:55 on dist-test-slave-nhpw"
server_key: "5888e79a7fb95e51562995c74bd07469"
server_key_iv: "421c6e200eae0c937d8fb694fb685001"
server_key_version: "encryptionkey@0"
I20260926 03:58:55.321779 8773 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 03:58:55.322431 8788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:55.322590 8773 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 03:58:55.322681 8773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-1/wal
uuid: "e91ee3ee2d844e298dd7f3a7e586502c"
format_stamp: "Formatted at 2026-09-26 03:58:55 on dist-test-slave-nhpw"
server_key: "5888e79a7fb95e51562995c74bd07469"
server_key_iv: "421c6e200eae0c937d8fb694fb685001"
server_key_version: "encryptionkey@0"
I20260926 03:58:55.322757 8773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:58:55.334388 8773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:58:55.336359 8773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:58:55.336526 8773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:58:55.336839 8773 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:58:55.337136 8773 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:58:55.337188 8773 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:55.337230 8773 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:58:55.337261 8773 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:55.342792 8773 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33309
I20260926 03:58:55.342859 8901 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33309 every 8 connection(s)
I20260926 03:58:55.343169 8773 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
Sep 26 03:58:55 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395135, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 03:58:55.349558 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 8773
I20260926 03:58:55.349660 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-1/wal/instance
I20260926 03:58:55.349879 392 external_mini_cluster.cc:1428] Setting key 72a2cdb05593747b7c03bfed61fa5e43
I20260926 03:58:55.353236 8636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:55.343902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44871 (local address 127.0.98.62:45737)
0926 03:58:55.343953 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:55.343956 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:55.344172 (+ 216us) server_negotiation.cc:407] Connection header received
0926 03:58:55.345178 (+ 1006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:55.345179 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:55.345189 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:55.345246 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:55.345833 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:55.346328 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:55.347320 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:55.347506 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:55.347563 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:55.349894 (+ 2331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:55.349895 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:55.349896 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:55.349902 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:55.351684 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:55.352189 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:55.352190 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:55.352191 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:55.352272 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:55.352557 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:55.352558 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:55.352559 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:55.352732 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:55.352802 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:55.353068 (+ 266us) server_negotiation.cc:299] Negotiation successful
0926 03:58:55.353173 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 03:58:55.353358 8904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:55.343889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45737 (local address 127.0.98.2:44871)
0926 03:58:55.344039 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:55.344074 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:55.345094 (+ 1020us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:55.345299 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:55.345302 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:55.345561 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:55.345765 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:55.345769 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:55.346424 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:55.346425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:55.347232 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:55.347235 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:55.347590 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:55.347591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:55.347688 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:55.348257 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:55.348265 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:55.349798 (+ 1533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:58:55.351767 (+ 1969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:55.351774 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:55.351777 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:55.352105 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:55.352374 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:55.352375 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:55.352376 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:55.352485 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:55.352806 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:58:55.352819 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:58:55.352939 (+ 120us) client_negotiation.cc:769] Sending connection context
0926 03:58:55.353028 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 03:58:55.353148 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":81,"threads_started":1}
I20260926 03:58:55.353767 8902 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45737
I20260926 03:58:55.353868 8902 heartbeater.cc:464] Registering TS with master...
I20260926 03:58:55.354048 8902 heartbeater.cc:511] Master 127.0.98.62:45737 requested a full tablet report, sending...
I20260926 03:58:55.354529 8577 ts_manager.cc:194] Registered new tserver with Master: e91ee3ee2d844e298dd7f3a7e586502c (127.0.98.2:33309)
I20260926 03:58:55.354966 8577 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44871
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 03:58:55.392006 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45737
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:58:55.485814 8907 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:58:55.486128 8907 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:58:55.486189 8907 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:58:55.487864 8907 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:58:55.487948 8907 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:58:55.488013 8907 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 03:58:55.490195 8907 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45737
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8907
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:58:55.490443 8907 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:58:55.490671 8907 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:58:55.496618 8913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:55.496598 8915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:55.496762 8907 server_base.cc:1034] running on GCE node
W20260926 03:58:55.496610 8912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:55.497066 8907 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:58:55.497345 8907 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:58:55.498497 8907 hybrid_clock.cc:648] HybridClock initialized: now 1790395135498477 us; error 30 us; skew 500 ppm
Sep 26 03:58:55 dist-test-slave-nhpw krb5kdc[8532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395135, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:58:55.500888 8907 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 03:58:55.501322 8907 webserver.cc:460] Webserver started at http://127.0.98.3:37993/ using document root <none> and password file <none>
I20260926 03:58:55.501506 8907 fs_manager.cc:359] Metadata directory not provided
I20260926 03:58:55.501544 8907 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:58:55.501627 8907 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:58:55.502580 8907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-2/data/instance:
uuid: "15807fbb78ef4922a030296bf384d248"
format_stamp: "Formatted at 2026-09-26 03:58:55 on dist-test-slave-nhpw"
server_key: "27ae671d63cec98c8d4c4a42c11a6c91"
server_key_iv: "4882967247d681a8858f752e6ac149dc"
server_key_version: "encryptionkey@0"
I20260926 03:58:55.502861 8907 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-2/wal/instance:
uuid: "15807fbb78ef4922a030296bf384d248"
format_stamp: "Formatted at 2026-09-26 03:58:55 on dist-test-slave-nhpw"
server_key: "27ae671d63cec98c8d4c4a42c11a6c91"
server_key_iv: "4882967247d681a8858f752e6ac149dc"
server_key_version: "encryptionkey@0"
I20260926 03:58:55.504034 8907 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 03:58:55.504714 8922 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:55.504891 8907 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:55.504977 8907 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-2/wal
uuid: "15807fbb78ef4922a030296bf384d248"
format_stamp: "Formatted at 2026-09-26 03:58:55 on dist-test-slave-nhpw"
server_key: "27ae671d63cec98c8d4c4a42c11a6c91"
server_key_iv: "4882967247d681a8858f752e6ac149dc"
server_key_version: "encryptionkey@0"
I20260926 03:58:55.505029 8907 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:58:55.527105 8907 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:58:55.529374 8907 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:58:55.529484 8907 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:58:55.529659 8907 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:58:55.530018 8907 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:58:55.530055 8907 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:55.530077 8907 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:58:55.530090 8907 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:55.535568 8907 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41061
I20260926 03:58:55.535615 9035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41061 every 8 connection(s)
I20260926 03:58:55.535909 8907 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
I20260926 03:58:55.536876 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 8907
I20260926 03:58:55.536990 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-2/wal/instance
I20260926 03:58:55.537153 392 external_mini_cluster.cc:1428] Setting key 0d844d3749e4e3a6a7666068eb3046bb
Sep 26 03:58:55 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395135, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 03:58:55.544924 8636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:55.536558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51473 (local address 127.0.98.62:45737)
0926 03:58:55.536625 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:55.536643 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:55.536897 (+ 254us) server_negotiation.cc:407] Connection header received
0926 03:58:55.538088 (+ 1191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:55.538089 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:55.538100 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:55.538145 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:55.538717 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:55.539212 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:55.540202 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:55.540406 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:55.540487 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:55.542453 (+ 1966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:55.542454 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:55.542455 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:55.542463 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:55.543546 (+ 1083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:55.543930 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:55.543932 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:55.543932 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:55.543986 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:55.544231 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:55.544231 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:55.544232 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:55.544380 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:55.544457 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:55.544754 (+ 297us) server_negotiation.cc:299] Negotiation successful
0926 03:58:55.544832 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 03:58:55.545184 9038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:55.536545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45737 (local address 127.0.98.3:51473)
0926 03:58:55.536778 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:55.536812 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:55.537986 (+ 1174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:55.538210 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:55.538214 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:55.538454 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:55.538630 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:55.538633 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:55.539312 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:55.539313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:55.540104 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:55.540106 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:55.540498 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:55.540499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:55.540622 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:55.541074 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:55.541082 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:55.542366 (+ 1284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:58:55.543624 (+ 1258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:55.543630 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:55.543632 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:55.543849 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:55.544048 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:58:55.544049 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:58:55.544049 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:58:55.544160 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:58:55.544447 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:58:55.544453 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:58:55.544588 (+ 135us) client_negotiation.cc:769] Sending connection context
0926 03:58:55.544737 (+ 149us) client_negotiation.cc:240] Negotiation successful
0926 03:58:55.544889 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":140,"threads_started":1}
I20260926 03:58:55.545679 9036 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45737
I20260926 03:58:55.545779 9036 heartbeater.cc:464] Registering TS with master...
I20260926 03:58:55.545959 9036 heartbeater.cc:511] Master 127.0.98.62:45737 requested a full tablet report, sending...
I20260926 03:58:55.546301 8577 ts_manager.cc:194] Registered new tserver with Master: 15807fbb78ef4922a030296bf384d248 (127.0.98.3:41061)
I20260926 03:58:55.546660 8577 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51473
I20260926 03:58:55.548679 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260926 03:58:55.673462 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 8639
I20260926 03:58:55.678105 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 8773
I20260926 03:58:55.682435 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 8907
I20260926 03:58:55.686647 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 8545
I20260926 03:58:55.691780 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45737
--webserver_interface=127.0.98.62
--webserver_port=45099
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45737
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:58:55.773363 9045 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:58:55.773568 9045 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:58:55.773591 9045 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:58:55.774785 9045 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:58:55.774827 9045 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:58:55.774842 9045 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:58:55.774854 9045 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:58:55.774866 9045 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:58:55.776583 9045 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45737
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=45099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9045
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:58:55.776839 9045 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:58:55.777033 9045 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:58:55.781795 9050 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:55.781795 9053 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:55.781855 9051 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:55.781905 9045 server_base.cc:1034] running on GCE node
I20260926 03:58:55.782224 9045 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:58:55.782546 9045 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:58:55.783715 9045 hybrid_clock.cc:648] HybridClock initialized: now 1790395135783693 us; error 40 us; skew 500 ppm
Sep 26 03:58:55 dist-test-slave-nhpw krb5kdc[8532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395135, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:58:55.786202 9045 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 03:58:55.786618 9045 webserver.cc:460] Webserver started at http://127.0.98.62:45099/ using document root <none> and password file <none>
I20260926 03:58:55.786832 9045 fs_manager.cc:359] Metadata directory not provided
I20260926 03:58:55.786911 9045 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:58:55.788252 9045 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 03:58:55.788906 9060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:55.789080 9045 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 03:58:55.789201 9045 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/master-0/wal
uuid: "4a97d90c240d4a3cae6ca86026d874cb"
format_stamp: "Formatted at 2026-09-26 03:58:54 on dist-test-slave-nhpw"
server_key: "4acc2b7d3b16856bd1f77ddc983d7907"
server_key_iv: "7ba2ef9371d3e6d4aa94ad80d2e2830d"
server_key_version: "encryptionkey@0"
I20260926 03:58:55.789529 9045 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:58:55.818303 9045 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:58:55.820312 9045 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:58:55.820482 9045 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:58:55.824276 9045 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45737
I20260926 03:58:55.824311 9112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45737 every 8 connection(s)
I20260926 03:58:55.824856 9045 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
I20260926 03:58:55.826076 9113 sys_catalog.cc:263] Verifying existing consensus state
I20260926 03:58:55.826320 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 9045
I20260926 03:58:55.826582 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:46801
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:45737
--webserver_port=46107
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 03:58:55.826784 9113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a97d90c240d4a3cae6ca86026d874cb: Bootstrap starting.
I20260926 03:58:55.834712 9113 log.cc:824] T 00000000000000000000000000000000 P 4a97d90c240d4a3cae6ca86026d874cb: Log is configured to *not* fsync() on all Append() calls
I20260926 03:58:55.836365 9113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a97d90c240d4a3cae6ca86026d874cb: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 03:58:55.836687 9113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a97d90c240d4a3cae6ca86026d874cb: Bootstrap complete.
I20260926 03:58:55.838884 9113 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a97d90c240d4a3cae6ca86026d874cb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a97d90c240d4a3cae6ca86026d874cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45737 } }
I20260926 03:58:55.839100 9113 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a97d90c240d4a3cae6ca86026d874cb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a97d90c240d4a3cae6ca86026d874cb, State: Initialized, Role: FOLLOWER
I20260926 03:58:55.839290 9113 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a97d90c240d4a3cae6ca86026d874cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a97d90c240d4a3cae6ca86026d874cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45737 } }
I20260926 03:58:55.839347 9113 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a97d90c240d4a3cae6ca86026d874cb [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:58:55.839402 9113 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a97d90c240d4a3cae6ca86026d874cb [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:58:55.839448 9113 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a97d90c240d4a3cae6ca86026d874cb [term 1 FOLLOWER]: Advancing to term 2
I20260926 03:58:55.840422 9113 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a97d90c240d4a3cae6ca86026d874cb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a97d90c240d4a3cae6ca86026d874cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45737 } }
I20260926 03:58:55.840507 9113 leader_election.cc:302] T 00000000000000000000000000000000 P 4a97d90c240d4a3cae6ca86026d874cb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a97d90c240d4a3cae6ca86026d874cb; no voters:
I20260926 03:58:55.840705 9113 leader_election.cc:288] T 00000000000000000000000000000000 P 4a97d90c240d4a3cae6ca86026d874cb [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 03:58:55.840760 9117 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a97d90c240d4a3cae6ca86026d874cb [term 2 FOLLOWER]: Leader election won for term 2
I20260926 03:58:55.840925 9117 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a97d90c240d4a3cae6ca86026d874cb [term 2 LEADER]: Becoming Leader. State: Replica: 4a97d90c240d4a3cae6ca86026d874cb, State: Running, Role: LEADER
I20260926 03:58:55.840967 9113 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a97d90c240d4a3cae6ca86026d874cb [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:58:55.841101 9117 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a97d90c240d4a3cae6ca86026d874cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a97d90c240d4a3cae6ca86026d874cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45737 } }
I20260926 03:58:55.841388 9119 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a97d90c240d4a3cae6ca86026d874cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a97d90c240d4a3cae6ca86026d874cb. Latest consensus state: current_term: 2 leader_uuid: "4a97d90c240d4a3cae6ca86026d874cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a97d90c240d4a3cae6ca86026d874cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45737 } } }
I20260926 03:58:55.841398 9118 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a97d90c240d4a3cae6ca86026d874cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4a97d90c240d4a3cae6ca86026d874cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a97d90c240d4a3cae6ca86026d874cb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45737 } } }
I20260926 03:58:55.841478 9119 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a97d90c240d4a3cae6ca86026d874cb [sys.catalog]: This master's current role is: LEADER
I20260926 03:58:55.841504 9118 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a97d90c240d4a3cae6ca86026d874cb [sys.catalog]: This master's current role is: LEADER
I20260926 03:58:55.841663 9123 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:58:55.842178 9123 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:58:55.842320 9123 catalog_manager.cc:1259] Loaded cluster ID: a1eba885c4ab4479af34cb12f1b6cb61
I20260926 03:58:55.842522 9123 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:58:55.843812 9123 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:58:55.844033 9123 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4a97d90c240d4a3cae6ca86026d874cb: Loaded TSK: 0
I20260926 03:58:55.844317 9123 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 03:58:55.909114 9115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:58:55.909350 9115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:58:55.909396 9115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:58:55.910559 9115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:58:55.910627 9115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:58:55.910698 9115 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 03:58:55.912209 9115 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:46801
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=46107
--tserver_master_addrs=127.0.98.62:45737
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:58:55.912453 9115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:58:55.912766 9115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:58:55.918834 9140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:55.918915 9115 server_base.cc:1034] running on GCE node
W20260926 03:58:55.918808 9142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:55.919013 9139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:55.919229 9115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:58:55.919473 9115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:58:55.920661 9115 hybrid_clock.cc:648] HybridClock initialized: now 1790395135920608 us; error 39 us; skew 500 ppm
Sep 26 03:58:55 dist-test-slave-nhpw krb5kdc[8532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395135, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:58:55.923333 9115 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260926 03:58:55.923892 9115 webserver.cc:460] Webserver started at http://127.0.98.1:46107/ using document root <none> and password file <none>
I20260926 03:58:55.924170 9115 fs_manager.cc:359] Metadata directory not provided
I20260926 03:58:55.924259 9115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:58:55.926167 9115 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 03:58:55.926838 9149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:55.927047 9115 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 03:58:55.927145 9115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-0/wal
uuid: "93d0ae3078024164a30f8ce74197d328"
format_stamp: "Formatted at 2026-09-26 03:58:55 on dist-test-slave-nhpw"
server_key: "d4f47c193b28b791479950d8784272b4"
server_key_iv: "3ca7372c393a9d4efb57fdfe8b31f41f"
server_key_version: "encryptionkey@0"
I20260926 03:58:55.927604 9115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:58:55.955348 9115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:58:55.957693 9115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:58:55.957849 9115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:58:55.958112 9115 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:58:55.958474 9115 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:58:55.958510 9115 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:55.958532 9115 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:58:55.958570 9115 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:55.965828 9115 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46801
I20260926 03:58:55.965952 9262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46801 every 8 connection(s)
I20260926 03:58:55.966267 9115 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
I20260926 03:58:55.971514 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 9115
I20260926 03:58:55.971853 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:33309
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:45737
--webserver_port=33947
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 03:58:55 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:58:55 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:58:55.975178 9265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:55.966996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45737 (local address 127.0.98.1:39563)
0926 03:58:55.967161 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:55.967202 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:55.967643 (+ 441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:55.968116 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:55.968120 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:55.968731 (+ 611us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:55.968920 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:55.968925 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:55.970282 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:55.970282 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:55.971050 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:55.971052 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:55.971578 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:55.971579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:55.971726 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:55.972285 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:55.972295 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:55.975031 (+ 2736us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45737: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":93,"threads_started":1}
W20260926 03:58:55.975369 9266 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:58:55.967060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39563 (local address 127.0.98.62:45737)
0926 03:58:55.967234 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:55.967242 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:55.967309 (+ 67us) server_negotiation.cc:407] Connection header received
0926 03:58:55.967864 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:55.967871 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:55.967925 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:55.968078 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:55.969034 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:55.970199 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:55.971125 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:55.971490 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:55.971568 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:55.975167 (+ 3599us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:39563: BlockingRecv error: recv got EOF from 127.0.98.1:39563 (error 108)
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":77,"threads_started":1}
W20260926 03:58:55.975410 9263 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45737 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:45737: Client connection negotiation failed: client connection to 127.0.98.62:45737: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 26 03:58:55 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:58:55 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:58:55.980183 9265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:55.975617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45737 (local address 127.0.98.1:51445)
0926 03:58:55.975667 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:55.975696 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:55.975744 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:55.975896 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:55.975897 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:55.976005 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:55.976056 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:55.976057 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:55.977055 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:55.977056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:55.977553 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:55.977554 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:55.978042 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:55.978042 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:55.978096 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:55.978566 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:55.978570 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:55.980141 (+ 1571us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45737: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 03:58:55.980330 9266 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:58:55.975732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51445 (local address 127.0.98.62:45737)
0926 03:58:55.975776 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:55.975778 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:55.975788 (+ 10us) server_negotiation.cc:407] Connection header received
0926 03:58:55.975811 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:55.975812 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:55.975825 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:55.975893 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:55.976127 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:55.976973 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:55.977627 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:55.977963 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:55.978045 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:55.980257 (+ 2212us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:51445: BlockingRecv error: recv got EOF from 127.0.98.1:51445 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 26 03:58:55 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:58:55 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:58:55.984607 9268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:55.980422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45737 (local address 127.0.98.1:58331)
0926 03:58:55.980581 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:55.980592 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:55.980661 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:55.980835 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:55.980836 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:55.980986 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:55.981026 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:55.981027 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:55.981605 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:55.981607 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:55.982139 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:55.982141 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:55.982504 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:55.982505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:55.982557 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:55.982985 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:55.982989 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:55.984513 (+ 1524us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45737: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":115,"threads_started":1}
W20260926 03:58:55.984671 9266 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:58:55.980669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58331 (local address 127.0.98.62:45737)
0926 03:58:55.980720 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:55.980722 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:55.980733 (+ 11us) server_negotiation.cc:407] Connection header received
0926 03:58:55.980750 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:55.980750 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:55.980763 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:55.980818 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:55.981105 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:55.981531 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:55.982232 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:55.982407 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:55.982474 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:55.984577 (+ 2103us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58331: BlockingRecv error: recv got EOF from 127.0.98.1:58331 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260926 03:58:55.984740 9263 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 03:58:56.055227 9267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:58:56.055474 9267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:58:56.055511 9267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:58:56.056711 9267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:58:56.056785 9267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:58:56.056829 9267 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 03:58:56.058276 9267 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:33309
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=33947
--tserver_master_addrs=127.0.98.62:45737
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9267
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:58:56.058547 9267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:58:56.058780 9267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:58:56.063745 9274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:56.063750 9276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:56.063732 9273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:56.064056 9267 server_base.cc:1034] running on GCE node
I20260926 03:58:56.064246 9267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:58:56.064441 9267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:58:56.065596 9267 hybrid_clock.cc:648] HybridClock initialized: now 1790395136065574 us; error 49 us; skew 500 ppm
Sep 26 03:58:56 dist-test-slave-nhpw krb5kdc[8532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395136, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:58:56.067999 9267 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260926 03:58:56.068437 9267 webserver.cc:460] Webserver started at http://127.0.98.2:33947/ using document root <none> and password file <none>
I20260926 03:58:56.068719 9267 fs_manager.cc:359] Metadata directory not provided
I20260926 03:58:56.068797 9267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:58:56.070137 9267 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 03:58:56.070770 9283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:56.070946 9267 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 03:58:56.071033 9267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-1/wal
uuid: "e91ee3ee2d844e298dd7f3a7e586502c"
format_stamp: "Formatted at 2026-09-26 03:58:55 on dist-test-slave-nhpw"
server_key: "5888e79a7fb95e51562995c74bd07469"
server_key_iv: "421c6e200eae0c937d8fb694fb685001"
server_key_version: "encryptionkey@0"
I20260926 03:58:56.071444 9267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:58:56.083842 9267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:58:56.086211 9267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:58:56.086330 9267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:58:56.086514 9267 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:58:56.086848 9267 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:58:56.086890 9267 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:56.086934 9267 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:58:56.086961 9267 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:56.092682 9267 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33309
I20260926 03:58:56.092743 9396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33309 every 8 connection(s)
I20260926 03:58:56.093138 9267 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
I20260926 03:58:56.097103 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 9267
I20260926 03:58:56.097417 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:41061
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:45737
--webserver_port=37993
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 03:58:56 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:58:56 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 03:58:56.100780 9266 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:58:56.093824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59507 (local address 127.0.98.62:45737)
0926 03:58:56.093867 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:56.093869 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:56.094076 (+ 207us) server_negotiation.cc:407] Connection header received
0926 03:58:56.094609 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:56.094610 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:56.094626 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:56.094674 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:56.095637 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:56.096153 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:56.097232 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:56.097432 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:56.097506 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:56.100732 (+ 3226us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:59507: BlockingRecv error: recv got EOF from 127.0.98.2:59507 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 03:58:56.100811 9399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:56.093803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45737 (local address 127.0.98.2:59507)
0926 03:58:56.093955 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:56.093989 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:56.094509 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:56.094760 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:56.094764 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:56.095344 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:56.095537 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:56.095543 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:56.096255 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:56.096256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:56.097133 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:56.097135 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:56.097511 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:56.097512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:56.097643 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:56.098152 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:56.098160 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:56.100550 (+ 2390us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45737: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":79,"threads_started":1}
W20260926 03:58:56.101404 9397 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45737 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:45737: Client connection negotiation failed: client connection to 127.0.98.62:45737: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 26 03:58:56 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:58:56 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:58:56.106642 9399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:56.101654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45737 (local address 127.0.98.2:39107)
0926 03:58:56.101710 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:56.101721 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:56.101760 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:56.101885 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:56.101886 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:56.102051 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:56.102114 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:56.102116 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:56.102752 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:56.102753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:56.103672 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:56.103675 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:56.104031 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:56.104032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:56.104109 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:56.104547 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:56.104553 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:56.106543 (+ 1990us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45737: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 03:58:56.106698 9266 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:58:56.101700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39107 (local address 127.0.98.62:45737)
0926 03:58:56.101756 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:56.101758 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:56.101768 (+ 10us) server_negotiation.cc:407] Connection header received
0926 03:58:56.101820 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:56.101820 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:56.101828 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:56.101878 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:56.102199 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:56.102674 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:56.103764 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:56.103947 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:56.104014 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:56.106636 (+ 2622us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:39107: BlockingRecv error: recv got EOF from 127.0.98.2:39107 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 26 03:58:56 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:58:56 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:58:56.111603 9399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:56.106892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45737 (local address 127.0.98.2:38883)
0926 03:58:56.107033 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:56.107043 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:56.107095 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:56.107229 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:56.107230 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:56.107385 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:56.107423 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:56.107424 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:56.108019 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:56.108020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:56.108904 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:56.108906 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:56.109261 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:56.109262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:56.109313 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:56.109732 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:56.109737 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:56.111515 (+ 1778us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45737: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":85,"threads_started":1}
W20260926 03:58:56.111663 9266 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:58:56.106968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38883 (local address 127.0.98.62:45737)
0926 03:58:56.107036 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:56.107038 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:56.107122 (+ 84us) server_negotiation.cc:407] Connection header received
0926 03:58:56.107145 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:56.107146 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:56.107154 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:56.107205 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:56.107493 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:56.107934 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:56.108982 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:56.109186 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:56.109253 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:56.111588 (+ 2335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:38883: BlockingRecv error: recv got EOF from 127.0.98.2:38883 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260926 03:58:56.111730 9397 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 03:58:56.181316 9400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:58:56.181545 9400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:58:56.181584 9400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:58:56.182785 9400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:58:56.182853 9400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:58:56.182899 9400 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 03:58:56.184355 9400 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:41061
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=37993
--tserver_master_addrs=127.0.98.62:45737
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9400
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:58:56.184623 9400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:58:56.184885 9400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:58:56.189805 9407 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:56.189796 9406 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:58:56.189796 9409 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:58:56.189963 9400 server_base.cc:1034] running on GCE node
I20260926 03:58:56.190258 9400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:58:56.190534 9400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:58:56.191673 9400 hybrid_clock.cc:648] HybridClock initialized: now 1790395136191661 us; error 30 us; skew 500 ppm
Sep 26 03:58:56 dist-test-slave-nhpw krb5kdc[8532](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395136, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:58:56.194235 9400 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260926 03:58:56.194623 9400 webserver.cc:460] Webserver started at http://127.0.98.3:37993/ using document root <none> and password file <none>
I20260926 03:58:56.194845 9400 fs_manager.cc:359] Metadata directory not provided
I20260926 03:58:56.194924 9400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:58:56.196308 9400 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 03:58:56.196964 9416 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:56.197125 9400 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 03:58:56.197233 9400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-2/wal
uuid: "15807fbb78ef4922a030296bf384d248"
format_stamp: "Formatted at 2026-09-26 03:58:55 on dist-test-slave-nhpw"
server_key: "27ae671d63cec98c8d4c4a42c11a6c91"
server_key_iv: "4882967247d681a8858f752e6ac149dc"
server_key_version: "encryptionkey@0"
I20260926 03:58:56.197577 9400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:58:56.209957 9400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:58:56.212137 9400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:58:56.212296 9400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:58:56.212505 9400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:58:56.212859 9400 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:58:56.212911 9400 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:56.212958 9400 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:58:56.212986 9400 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:58:56.218654 9400 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41061
I20260926 03:58:56.218729 9529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41061 every 8 connection(s)
I20260926 03:58:56.219051 9400 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
I20260926 03:58:56.222419 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 9400
Sep 26 03:58:56 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:58:56 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 03:58:56.226701 9266 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:58:56.219792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38157 (local address 127.0.98.62:45737)
0926 03:58:56.219841 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:56.219843 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:56.220043 (+ 200us) server_negotiation.cc:407] Connection header received
0926 03:58:56.220525 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:56.220526 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:56.220536 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:56.220582 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:56.221520 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:56.222006 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:56.223095 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:56.223355 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:56.223407 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:56.226655 (+ 3248us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:38157: BlockingRecv error: recv got EOF from 127.0.98.3:38157 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 03:58:56.226791 9532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:56.219783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45737 (local address 127.0.98.3:38157)
0926 03:58:56.219922 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:56.219958 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:56.220441 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:56.220685 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:56.220689 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:56.221252 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:56.221444 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:56.221448 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:56.222078 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:56.222079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:56.223020 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:56.223023 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:56.223430 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:56.223431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:56.223590 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:56.224133 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:56.224141 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:56.226587 (+ 2446us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45737: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":74,"threads_started":1}
W20260926 03:58:56.226974 9530 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:45737 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:45737: Client connection negotiation failed: client connection to 127.0.98.62:45737: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
I20260926 03:58:56.229938 9534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:56.222924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44976 (local address 127.0.98.62:45737)
0926 03:58:56.223028 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:56.223030 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:56.223038 (+ 8us) server_negotiation.cc:407] Connection header received
0926 03:58:56.223095 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:56.223095 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:56.223106 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:56.223171 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:56.223855 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:56.224327 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:56.225199 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:56.225452 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:56.225527 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:56.226885 (+ 1358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:58:56.226890 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:58:56.226890 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:58:56.226901 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:58:56.228573 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:56.228961 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:56.228964 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:56.228965 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:56.229056 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:58:56.229255 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:58:56.229256 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:58:56.229257 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:58:56.229468 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:58:56.229605 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 03:58:56.229713 (+ 108us) server_negotiation.cc:299] Negotiation successful
0926 03:58:56.229821 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":62,"threads_started":1}
Sep 26 03:58:56 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:58:56 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:58:56.231395 9532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:56.227219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45737 (local address 127.0.98.3:47391)
0926 03:58:56.227250 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:56.227260 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:56.227298 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:56.227424 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:56.227424 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:56.227544 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:56.227581 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:56.227583 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:56.228178 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:56.228179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:56.228723 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:56.228724 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:56.229068 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:56.229068 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:56.229132 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:56.229503 (+ 371us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:56.229508 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:56.231321 (+ 1813us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45737: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":16}
W20260926 03:58:56.231441 9266 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:58:56.227253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47391 (local address 127.0.98.62:45737)
0926 03:58:56.227292 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:56.227294 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:56.227302 (+ 8us) server_negotiation.cc:407] Connection header received
0926 03:58:56.227357 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:56.227357 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:56.227367 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:56.227420 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:56.227651 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:56.228090 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:56.228808 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:56.228990 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:56.229067 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:56.231384 (+ 2317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:47391: BlockingRecv error: recv got EOF from 127.0.98.3:47391 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
Sep 26 03:58:56 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:58:56 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:58:56.235160 9532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:56.231677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45737 (local address 127.0.98.3:55513)
0926 03:58:56.231724 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:56.231735 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:56.231778 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:56.231884 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:56.231884 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:56.231984 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:56.232007 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:56.232008 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:56.232518 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:56.232519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:56.233030 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:56.233032 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:56.233333 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:56.233333 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:56.233368 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:56.233674 (+ 306us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:56.233677 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:56.235119 (+ 1442us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45737: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20260926 03:58:56.235220 9266 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:58:56.231736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55513 (local address 127.0.98.62:45737)
0926 03:58:56.231777 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:56.231778 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:56.231785 (+ 7us) server_negotiation.cc:407] Connection header received
0926 03:58:56.231834 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:56.231835 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:56.231842 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:56.231882 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:56.232048 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:56.232464 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:56.233094 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:56.233271 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:56.233324 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:56.235187 (+ 1863us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55513: BlockingRecv error: recv got EOF from 127.0.98.3:55513 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260926 03:58:56.235308 9530 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 26 03:58:56 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:58:56 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:58:56.989620 9535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:56.985083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45737 (local address 127.0.98.1:49519)
0926 03:58:56.985222 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:56.985232 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:56.985285 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:56.985405 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:56.985406 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:56.985566 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:56.985631 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:56.985632 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:56.986244 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:56.986245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:56.986780 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:56.986782 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:56.987181 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:56.987183 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:56.987284 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:56.987793 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:56.987798 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:56.989514 (+ 1716us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45737: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":80,"threads_started":1}
W20260926 03:58:56.989665 9536 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:58:56.985158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49519 (local address 127.0.98.62:45737)
0926 03:58:56.985296 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:56.985298 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:56.985306 (+ 8us) server_negotiation.cc:407] Connection header received
0926 03:58:56.985326 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:56.985327 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:56.985336 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:56.985393 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:56.985691 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:56.986183 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:56.986873 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:56.987061 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:56.987129 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:56.989592 (+ 2463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:49519: BlockingRecv error: recv got EOF from 127.0.98.1:49519 (error 108)
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":73,"threads_started":1}
Sep 26 03:58:57 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:58:57 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:58:57.117141 9537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:57.112172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45737 (local address 127.0.98.2:33255)
0926 03:58:57.112336 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:57.112348 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:57.112413 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:57.112604 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:57.112606 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:57.112846 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:57.112896 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:57.112898 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:57.113600 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:57.113601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:57.114171 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:57.114172 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:57.114581 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:57.114582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:57.114627 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:57.115159 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:57.115164 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:57.117069 (+ 1905us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45737: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":74,"threads_started":1}
W20260926 03:58:57.117254 9536 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:58:57.112272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33255 (local address 127.0.98.62:45737)
0926 03:58:57.112338 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:57.112341 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:57.112409 (+ 68us) server_negotiation.cc:407] Connection header received
0926 03:58:57.112484 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:57.112485 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:57.112500 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:57.112581 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:57.113013 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:57.113515 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:57.114307 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:57.114504 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:57.114570 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:57.117170 (+ 2600us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:33255: BlockingRecv error: recv got EOF from 127.0.98.2:33255 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 26 03:58:57 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:58:57 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:58:57.240487 9538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:57.235720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45737 (local address 127.0.98.3:44871)
0926 03:58:57.235910 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:57.235923 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:57.236013 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:57.236187 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:57.236188 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:57.236369 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:57.236416 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:57.236418 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:57.237119 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:57.237120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:57.237661 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:57.237663 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:57.238040 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:57.238041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:57.238104 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:57.238581 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:57.238585 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:57.240382 (+ 1797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45737: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":102,"threads_started":1}
W20260926 03:58:57.240527 9536 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:58:57.235833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44871 (local address 127.0.98.62:45737)
0926 03:58:57.235913 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:57.235916 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:57.235994 (+ 78us) server_negotiation.cc:407] Connection header received
0926 03:58:57.236103 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:57.236104 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:57.236122 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:57.236187 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:57.236497 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:57.237038 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:57.237761 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:57.237952 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:57.238037 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:57.240460 (+ 2423us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:44871: BlockingRecv error: recv got EOF from 127.0.98.3:44871 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 26 03:58:57 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:58:57 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:58:57.994717 9539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:57.990096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45737 (local address 127.0.98.1:47147)
0926 03:58:57.990279 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:57.990289 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:57.990351 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:57.990485 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:57.990486 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:57.990643 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:57.990690 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:57.990691 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:57.991343 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:57.991344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:57.991920 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:57.991923 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:57.992314 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:57.992315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:57.992357 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:57.992971 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:57.992976 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:57.994658 (+ 1682us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45737: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":94,"threads_started":1}
W20260926 03:58:57.994946 9540 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:58:57.990158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47147 (local address 127.0.98.62:45737)
0926 03:58:57.990276 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:57.990278 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:57.990356 (+ 78us) server_negotiation.cc:407] Connection header received
0926 03:58:57.990406 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:57.990406 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:57.990417 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:57.990467 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:57.990747 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:57.991250 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:57.991998 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:57.992231 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:57.992316 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:57.994759 (+ 2443us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47147: BlockingRecv error: recv got EOF from 127.0.98.1:47147 (error 108)
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":46,"threads_started":1}
Sep 26 03:58:58 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:58:58 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:58:58.122586 9541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:58.117565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45737 (local address 127.0.98.2:34075)
0926 03:58:58.117763 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:58.117776 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:58.117862 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:58.118005 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:58.118005 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:58.118173 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:58.118219 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:58.118221 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:58.118912 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:58.118913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:58.119447 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:58.119449 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:58.120082 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:58.120084 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:58.120153 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:58.120624 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:58.120629 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:58.122518 (+ 1889us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45737: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":92,"threads_started":1}
W20260926 03:58:58.122740 9540 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:58:58.117666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34075 (local address 127.0.98.62:45737)
0926 03:58:58.117758 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:58.117767 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:58.117844 (+ 77us) server_negotiation.cc:407] Connection header received
0926 03:58:58.117925 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:58.117926 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:58.117939 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:58.117994 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:58.118317 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:58.118830 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:58.119531 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:58.119952 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:58.120075 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:58.122638 (+ 2563us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:34075: BlockingRecv error: recv got EOF from 127.0.98.2:34075 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 26 03:58:58 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:58:58 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:58:58.245780 9542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:58.240977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45737 (local address 127.0.98.3:45819)
0926 03:58:58.241172 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:58.241184 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:58.241265 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:58.241532 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:58.241533 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:58.241711 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:58.241757 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:58.241759 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:58.242463 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:58.242464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:58.243071 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:58.243073 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:58.243395 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:58.243396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:58.243437 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:58.243917 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:58.243922 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:58.245718 (+ 1796us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45737: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":73,"threads_started":1}
W20260926 03:58:58.245895 9540 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:58:58.241153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45819 (local address 127.0.98.62:45737)
0926 03:58:58.241217 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:58.241247 (+ 30us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:58.241258 (+ 11us) server_negotiation.cc:407] Connection header received
0926 03:58:58.241333 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:58.241333 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:58.241347 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:58.241390 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:58.241846 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:58.242377 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:58.243142 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:58.243328 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:58.243380 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:58.245796 (+ 2416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:45819: BlockingRecv error: recv got EOF from 127.0.98.3:45819 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 26 03:58:58 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:58:58 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:58:59.000201 9543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:58.995369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45737 (local address 127.0.98.1:41733)
0926 03:58:58.995488 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:58.995502 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:58.995591 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:58.995743 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:58.995744 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:58.995899 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:58.995948 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:58.995949 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:58.996803 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:58.996805 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:58.997541 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:58.997544 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:58.997868 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:58.997868 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:58.997908 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:58.998353 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:58.998358 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:59.000099 (+ 1741us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45737: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":56,"threads_started":1}
W20260926 03:58:59.000247 9544 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:58:58.995447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41733 (local address 127.0.98.62:45737)
0926 03:58:58.995596 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:58.995598 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:58.995641 (+ 43us) server_negotiation.cc:407] Connection header received
0926 03:58:58.995664 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:58.995664 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:58.995674 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:58.995725 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:58.996018 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:58.996583 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:58.997605 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:58.997805 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:58.997859 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:59.000180 (+ 2321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:41733: BlockingRecv error: recv got EOF from 127.0.98.1:41733 (error 108)
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":83,"threads_started":1}
Sep 26 03:58:59 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:58:59 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:58:59.128244 9545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:59.123166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45737 (local address 127.0.98.2:36807)
0926 03:58:59.123318 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:59.123331 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:59.123404 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:59.123581 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:59.123582 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:59.123761 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:59.123815 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:59.123817 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:59.124483 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:59.124484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:59.125062 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:59.125065 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:59.125463 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:59.125465 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:59.125525 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:59.126009 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:59.126013 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:59.128183 (+ 2170us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45737: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":104,"threads_started":1}
W20260926 03:58:59.128373 9544 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:58:59.123237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36807 (local address 127.0.98.62:45737)
0926 03:58:59.123329 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:59.123330 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:59.123408 (+ 78us) server_negotiation.cc:407] Connection header received
0926 03:58:59.123456 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:59.123456 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:59.123467 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:59.123527 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:59.123896 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:59.124397 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:59.125166 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:59.125370 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:59.125436 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:59.128299 (+ 2863us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:36807: BlockingRecv error: recv got EOF from 127.0.98.2:36807 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 26 03:58:59 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:58:59 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:58:59.250633 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:58:59.246239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45737 (local address 127.0.98.3:33017)
0926 03:58:59.246371 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 03:58:59.246386 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 03:58:59.246438 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:58:59.246623 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:58:59.246624 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:58:59.246836 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:58:59.246901 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:59.246903 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:59.247563 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:59.247564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:59.248099 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:58:59.248100 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:58:59.248408 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:58:59.248409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:58:59.248451 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:59.248901 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:58:59.248905 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:58:59.250576 (+ 1671us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45737: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":80,"threads_started":1}
W20260926 03:58:59.250765 9544 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:58:59.246295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33017 (local address 127.0.98.62:45737)
0926 03:58:59.246463 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 03:58:59.246465 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:58:59.246506 (+ 41us) server_negotiation.cc:407] Connection header received
0926 03:58:59.246531 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:58:59.246531 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:58:59.246557 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:58:59.246610 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:58:59.246988 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:59.247493 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:59.248172 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:58:59.248349 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:58:59.248399 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:58:59.250672 (+ 2273us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:33017: BlockingRecv error: recv got EOF from 127.0.98.3:33017 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 26 03:58:59 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:59:00 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:59:00.005620 9547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:00.000689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45737 (local address 127.0.98.1:35413)
0926 03:59:00.000867 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 03:59:00.000877 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 03:59:00.000986 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:59:00.001220 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:59:00.001221 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:59:00.001428 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:59:00.001477 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:00.001479 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:00.002127 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:00.002128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:00.002657 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:00.002659 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:00.002979 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:00.002979 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:00.003023 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:00.003532 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:59:00.003538 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:59:00.005510 (+ 1972us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45737: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":119,"threads_started":1}
W20260926 03:59:00.005671 9548 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:59:00.000745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35413 (local address 127.0.98.62:45737)
0926 03:59:00.000869 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 03:59:00.000872 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 03:59:00.000979 (+ 107us) server_negotiation.cc:407] Connection header received
0926 03:59:00.001079 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:59:00.001080 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:59:00.001090 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:59:00.001164 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:59:00.001544 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:00.002052 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:00.002727 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:00.002915 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:00.002966 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:00.005598 (+ 2632us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35413: BlockingRecv error: recv got EOF from 127.0.98.1:35413 (error 108)
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":63,"threads_started":1}
Sep 26 03:59:00 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:59:00 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:59:00.133033 9549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:00.128787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45737 (local address 127.0.98.2:36393)
0926 03:59:00.128893 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0926 03:59:00.128910 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:59:00.128967 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:59:00.129079 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:59:00.129079 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:59:00.129272 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:59:00.129325 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:00.129328 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:00.129947 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:00.129947 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:00.130465 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:00.130466 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:00.130763 (+ 297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:00.130764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:00.130801 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:00.131322 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:59:00.131327 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:59:00.132975 (+ 1648us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45737: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":50,"threads_started":1}
W20260926 03:59:00.133139 9548 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:59:00.128841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36393 (local address 127.0.98.62:45737)
0926 03:59:00.128915 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 03:59:00.128917 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:59:00.128952 (+ 35us) server_negotiation.cc:407] Connection header received
0926 03:59:00.129020 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:59:00.129021 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:59:00.129031 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:59:00.129065 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:59:00.129385 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:00.129883 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:00.130529 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:00.130703 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:00.130761 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:00.133067 (+ 2306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:36393: BlockingRecv error: recv got EOF from 127.0.98.2:36393 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 26 03:59:00 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:59:00 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:59:00.255658 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:00.251164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45737 (local address 127.0.98.3:53553)
0926 03:59:00.251322 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 03:59:00.251333 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 03:59:00.251394 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:59:00.251522 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:59:00.251523 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:59:00.251680 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:59:00.251742 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:00.251744 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:00.252373 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:00.252373 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:00.252956 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:00.252958 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:00.253272 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:00.253273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:00.253312 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:00.253808 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:59:00.253812 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:59:00.255558 (+ 1746us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45737: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":71,"threads_started":1}
W20260926 03:59:00.255704 9548 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:59:00.251214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53553 (local address 127.0.98.62:45737)
0926 03:59:00.251320 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0926 03:59:00.251322 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:59:00.251392 (+ 70us) server_negotiation.cc:407] Connection header received
0926 03:59:00.251451 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:59:00.251452 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:59:00.251464 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:59:00.251504 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:59:00.251800 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:00.252301 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:00.253031 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:00.253202 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:00.253263 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:00.255639 (+ 2376us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:53553: BlockingRecv error: recv got EOF from 127.0.98.3:53553 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 26 03:59:01 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:59:01 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:59:01.013963 9551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:01.006150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45737 (local address 127.0.98.1:39335)
0926 03:59:01.006307 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 03:59:01.006317 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 03:59:01.006370 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:59:01.006495 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:59:01.006496 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:59:01.006651 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:59:01.006702 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:01.006704 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:01.007413 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:01.007414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:01.007959 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:01.007961 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:01.008311 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:01.008311 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:01.008350 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:01.012170 (+ 3820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:59:01.012175 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:59:01.013904 (+ 1729us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45737: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":103,"threads_started":1}
W20260926 03:59:01.014096 9552 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:59:01.006245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39335 (local address 127.0.98.62:45737)
0926 03:59:01.006357 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 03:59:01.006359 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:59:01.006368 (+ 9us) server_negotiation.cc:407] Connection header received
0926 03:59:01.006415 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:59:01.006415 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:59:01.006425 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:59:01.006484 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:59:01.006790 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:01.007334 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:01.008040 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:01.008232 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:01.008310 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:01.013998 (+ 5688us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:39335: BlockingRecv error: recv got EOF from 127.0.98.1:39335 (error 108)
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":50,"threads_started":1}
Sep 26 03:59:01 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:59:01 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:59:01.137830 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:01.133493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45737 (local address 127.0.98.2:56729)
0926 03:59:01.133637 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 03:59:01.133647 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 03:59:01.133698 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:59:01.133818 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:59:01.133819 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:59:01.133986 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:59:01.134036 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:01.134037 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:01.134669 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:01.134670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:01.135201 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:01.135203 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:01.135538 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:01.135539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:01.135585 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:01.136112 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:59:01.136116 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:59:01.137775 (+ 1659us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45737: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":79,"threads_started":1}
W20260926 03:59:01.137920 9552 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:59:01.133616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56729 (local address 127.0.98.62:45737)
0926 03:59:01.133685 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 03:59:01.133686 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 03:59:01.133695 (+ 9us) server_negotiation.cc:407] Connection header received
0926 03:59:01.133760 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:59:01.133761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:59:01.133771 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:59:01.133808 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:59:01.134090 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:01.134579 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:01.135273 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:01.135447 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:01.135498 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:01.137856 (+ 2358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:56729: BlockingRecv error: recv got EOF from 127.0.98.2:56729 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 26 03:59:01 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:59:01 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:59:01.261149 9554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:01.256152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45737 (local address 127.0.98.3:35827)
0926 03:59:01.256309 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 03:59:01.256326 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 03:59:01.256411 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:59:01.256588 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:59:01.256589 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:59:01.256864 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:59:01.256922 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:01.256925 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:01.257687 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:01.257688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:01.258296 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:01.258297 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:01.258704 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:01.258705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:01.258770 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:01.259296 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:59:01.259302 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:59:01.261081 (+ 1779us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45737: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":77,"threads_started":1}
W20260926 03:59:01.261258 9552 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:59:01.256256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35827 (local address 127.0.98.62:45737)
0926 03:59:01.256333 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 03:59:01.256336 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 03:59:01.256388 (+ 52us) server_negotiation.cc:407] Connection header received
0926 03:59:01.256484 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:59:01.256485 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:59:01.256501 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:59:01.256582 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:59:01.257010 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:01.257592 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:01.258397 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:01.258610 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:01.258689 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:01.261183 (+ 2494us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:35827: BlockingRecv error: recv got EOF from 127.0.98.3:35827 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 26 03:59:02 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:59:02 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:59:02.019297 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:02.014539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45737 (local address 127.0.98.1:46797)
0926 03:59:02.014722 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 03:59:02.014732 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 03:59:02.014813 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:59:02.014975 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:59:02.014976 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:59:02.015154 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:59:02.015229 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:02.015231 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:02.015907 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:02.015908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:02.016456 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:02.016458 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:02.016889 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:02.016890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:02.016948 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:02.017454 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:59:02.017460 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:59:02.019192 (+ 1732us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45737: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":73,"threads_started":1}
W20260926 03:59:02.019351 9556 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:59:02.014570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46797 (local address 127.0.98.62:45737)
0926 03:59:02.014723 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 03:59:02.014726 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 03:59:02.014794 (+ 68us) server_negotiation.cc:407] Connection header received
0926 03:59:02.014875 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:59:02.014876 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:59:02.014890 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:59:02.014968 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:59:02.015307 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:02.015826 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:02.016539 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:02.016806 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:02.016887 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:02.019276 (+ 2389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46797: BlockingRecv error: recv got EOF from 127.0.98.1:46797 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":58,"threads_started":1}
Sep 26 03:59:02 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:59:02 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:59:02.143023 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:02.138321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45737 (local address 127.0.98.2:42061)
0926 03:59:02.138497 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 03:59:02.138510 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 03:59:02.138585 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:59:02.138745 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:59:02.138745 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:59:02.138930 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:59:02.138977 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:02.138979 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:02.139651 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:02.139653 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:02.140198 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:02.140200 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:02.140562 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:02.140563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:02.140619 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:02.141170 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:59:02.141175 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:59:02.142925 (+ 1750us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45737: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":89,"threads_started":1}
W20260926 03:59:02.143100 9556 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:59:02.138425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42061 (local address 127.0.98.62:45737)
0926 03:59:02.138505 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0926 03:59:02.138508 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 03:59:02.138573 (+ 65us) server_negotiation.cc:407] Connection header received
0926 03:59:02.138652 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:59:02.138653 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:59:02.138666 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:59:02.138739 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:59:02.139054 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:02.139570 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:02.140286 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:02.140478 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:02.140562 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:02.143031 (+ 2469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42061: BlockingRecv error: recv got EOF from 127.0.98.2:42061 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 26 03:59:02 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:59:02 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:59:02.267099 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:02.261736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45737 (local address 127.0.98.3:38627)
0926 03:59:02.262034 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0926 03:59:02.262045 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 03:59:02.262114 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:59:02.262266 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:59:02.262267 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:59:02.262425 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:59:02.262475 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:02.262477 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:02.263374 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:02.263375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:02.263901 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:02.263903 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:02.264623 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:02.264624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:02.264726 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:02.265262 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:59:02.265268 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:59:02.267036 (+ 1768us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45737: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":211,"threads_started":1}
W20260926 03:59:02.267230 9556 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:59:02.261824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38627 (local address 127.0.98.62:45737)
0926 03:59:02.261932 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0926 03:59:02.261935 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 03:59:02.262122 (+ 187us) server_negotiation.cc:407] Connection header received
0926 03:59:02.262153 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:59:02.262154 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:59:02.262168 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:59:02.262231 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:59:02.262553 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:02.263268 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:02.264175 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:02.264498 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:02.264593 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:02.267130 (+ 2537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:38627: BlockingRecv error: recv got EOF from 127.0.98.3:38627 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 26 03:59:03 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:59:03 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:59:03.024662 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:03.019847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45737 (local address 127.0.98.1:40833)
0926 03:59:03.020018 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 03:59:03.020036 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 03:59:03.020097 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:59:03.020231 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:59:03.020231 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:59:03.020413 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:59:03.020461 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:03.020462 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:03.021232 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:03.021232 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:03.021772 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:03.021774 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:03.022111 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:03.022112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:03.022148 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:03.022607 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:59:03.022611 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:59:03.024545 (+ 1934us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45737: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":93,"threads_started":1}
W20260926 03:59:03.024710 9560 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:59:03.019917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40833 (local address 127.0.98.62:45737)
0926 03:59:03.020044 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 03:59:03.020046 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:59:03.020084 (+ 38us) server_negotiation.cc:407] Connection header received
0926 03:59:03.020144 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:59:03.020145 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:59:03.020155 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:59:03.020219 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:59:03.020535 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:03.021160 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:03.021843 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:03.022048 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:03.022104 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:03.024617 (+ 2513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:40833: BlockingRecv error: recv got EOF from 127.0.98.1:40833 (error 108)
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":79,"threads_started":1}
Sep 26 03:59:03 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:59:03 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:59:03.148392 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:03.143611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45737 (local address 127.0.98.2:34729)
0926 03:59:03.143719 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0926 03:59:03.143735 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:59:03.143815 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:59:03.143991 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:59:03.143992 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:59:03.144165 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:59:03.144212 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:03.144214 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:03.144956 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:03.144957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:03.145591 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:03.145593 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:03.145968 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:03.145969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:03.146047 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:03.146544 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:59:03.146548 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:59:03.148295 (+ 1747us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45737: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":63,"threads_started":1}
W20260926 03:59:03.148437 9560 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:59:03.143637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34729 (local address 127.0.98.62:45737)
0926 03:59:03.143701 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 03:59:03.143703 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:59:03.143802 (+ 99us) server_negotiation.cc:407] Connection header received
0926 03:59:03.143890 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:59:03.143891 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:59:03.143905 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:59:03.143972 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:59:03.144304 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:03.144876 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:03.145694 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:03.145897 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:03.145971 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:03.148373 (+ 2402us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:34729: BlockingRecv error: recv got EOF from 127.0.98.2:34729 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 26 03:59:03 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:59:03 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:59:03.271914 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:03.267541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45737 (local address 127.0.98.3:60265)
0926 03:59:03.267706 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 03:59:03.267718 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 03:59:03.267783 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:59:03.267932 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:59:03.267933 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:59:03.268084 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:59:03.268133 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:03.268135 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:03.268820 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:03.268821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:03.269335 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:03.269336 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:03.269640 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:03.269641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:03.269679 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:03.270118 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:59:03.270122 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:59:03.271857 (+ 1735us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45737: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":97,"threads_started":1}
W20260926 03:59:03.272004 9560 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:59:03.267610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60265 (local address 127.0.98.62:45737)
0926 03:59:03.267691 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 03:59:03.267693 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:59:03.267774 (+ 81us) server_negotiation.cc:407] Connection header received
0926 03:59:03.267847 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:59:03.267848 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:59:03.267860 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:59:03.267913 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:59:03.268216 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:03.268748 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:03.269403 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:03.269581 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:03.269630 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:03.271939 (+ 2309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:60265: BlockingRecv error: recv got EOF from 127.0.98.3:60265 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 26 03:59:04 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:59:04 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:59:04.030162 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:04.025207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45737 (local address 127.0.98.1:36429)
0926 03:59:04.025369 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 03:59:04.025380 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 03:59:04.025430 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:59:04.025566 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:59:04.025567 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:59:04.025732 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:59:04.025782 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:04.025783 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:04.026465 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:04.026466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:04.027070 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:04.027071 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:04.027479 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:04.027479 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:04.027529 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:04.028163 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:59:04.028169 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:59:04.030065 (+ 1896us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45737: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":102,"threads_started":1}
W20260926 03:59:04.030300 9564 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:59:04.025325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36429 (local address 127.0.98.62:45737)
0926 03:59:04.025451 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 03:59:04.025453 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:59:04.025461 (+ 8us) server_negotiation.cc:407] Connection header received
0926 03:59:04.025483 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:59:04.025483 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:59:04.025492 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:59:04.025551 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:59:04.025846 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:04.026369 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:04.027155 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:04.027395 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:04.027455 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:04.030163 (+ 2708us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:36429: BlockingRecv error: recv got EOF from 127.0.98.1:36429 (error 108)
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":74,"threads_started":1}
Sep 26 03:59:04 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:59:04 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:59:04.153666 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:04.148953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45737 (local address 127.0.98.2:35905)
0926 03:59:04.149110 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 03:59:04.149126 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:59:04.149184 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:59:04.149367 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:59:04.149368 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:59:04.149527 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:59:04.149578 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:04.149579 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:04.150390 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:04.150391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:04.150931 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:04.150932 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:04.151246 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:04.151246 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:04.151285 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:04.151714 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:59:04.151719 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:59:04.153567 (+ 1848us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45737: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":99,"threads_started":1}
W20260926 03:59:04.153712 9564 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:59:04.149069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35905 (local address 127.0.98.62:45737)
0926 03:59:04.149147 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 03:59:04.149149 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:59:04.149174 (+ 25us) server_negotiation.cc:407] Connection header received
0926 03:59:04.149278 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:59:04.149279 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:59:04.149289 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:59:04.149352 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:59:04.149650 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:04.150310 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:04.151002 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:04.151186 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:04.151236 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:04.153648 (+ 2412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35905: BlockingRecv error: recv got EOF from 127.0.98.2:35905 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 26 03:59:04 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:59:04 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:59:04.277125 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:04.272435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45737 (local address 127.0.98.3:52753)
0926 03:59:04.272572 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 03:59:04.272582 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 03:59:04.272681 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:59:04.272807 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:59:04.272808 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:59:04.272955 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:59:04.273010 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:04.273012 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:04.273698 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:04.273698 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:04.274290 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:04.274293 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:04.274783 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:04.274784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:04.274849 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:04.275302 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:59:04.275307 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:59:04.277069 (+ 1762us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45737: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":66,"threads_started":1}
W20260926 03:59:04.277225 9564 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:59:04.272501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52753 (local address 127.0.98.62:45737)
0926 03:59:04.272587 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0926 03:59:04.272589 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:59:04.272667 (+ 78us) server_negotiation.cc:407] Connection header received
0926 03:59:04.272732 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:59:04.272733 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:59:04.272745 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:59:04.272794 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:59:04.273068 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:04.273577 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:04.274374 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:04.274701 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:04.274763 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:04.277153 (+ 2390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:52753: BlockingRecv error: recv got EOF from 127.0.98.3:52753 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 26 03:59:05 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:59:05 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:59:05.041211 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:05.030693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45737 (local address 127.0.98.1:33911)
0926 03:59:05.030850 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 03:59:05.030869 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 03:59:05.030917 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:59:05.031149 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:59:05.031150 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:59:05.031339 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:59:05.031400 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:05.031402 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:05.032160 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:05.032161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:05.032737 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:05.032738 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:05.033159 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:05.033160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:05.033232 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:05.033825 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:59:05.033831 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:59:05.041094 (+ 7263us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45737: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":103,"threads_started":1}
W20260926 03:59:05.041567 9568 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:59:05.030819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33911 (local address 127.0.98.62:45737)
0926 03:59:05.030955 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 03:59:05.030956 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 03:59:05.030986 (+ 30us) server_negotiation.cc:407] Connection header received
0926 03:59:05.031008 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:59:05.031009 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:59:05.031018 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:59:05.031094 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:59:05.031497 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:05.032082 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:05.032804 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:05.033055 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:05.033125 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:05.041497 (+ 8372us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33911: BlockingRecv error: recv got EOF from 127.0.98.1:33911 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":81,"threads_started":1}
Sep 26 03:59:05 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:59:05 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:59:05.158666 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:05.154209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45737 (local address 127.0.98.2:56045)
0926 03:59:05.154370 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 03:59:05.154380 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 03:59:05.154441 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:59:05.154554 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:59:05.154554 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:59:05.154699 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:59:05.154748 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:05.154749 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:05.155378 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:05.155379 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:05.155907 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:05.155909 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:05.156258 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:05.156259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:05.156309 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:05.156835 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:59:05.156840 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:59:05.158608 (+ 1768us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45737: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":121,"threads_started":1}
W20260926 03:59:05.158823 9568 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:59:05.154269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56045 (local address 127.0.98.62:45737)
0926 03:59:05.154343 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 03:59:05.154345 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:59:05.154441 (+ 96us) server_negotiation.cc:407] Connection header received
0926 03:59:05.154499 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:59:05.154500 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:59:05.154510 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:59:05.154548 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:59:05.154802 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:05.155304 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:05.155987 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:05.156176 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:05.156228 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:05.158727 (+ 2499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:56045: BlockingRecv error: recv got EOF from 127.0.98.2:56045 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 26 03:59:05 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:59:05 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 03:59:05.282630 9568 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:59:05.277686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34451 (local address 127.0.98.62:45737)
0926 03:59:05.277754 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 03:59:05.277756 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:59:05.277802 (+ 46us) server_negotiation.cc:407] Connection header received
0926 03:59:05.277869 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:59:05.277870 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:59:05.277881 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:59:05.277938 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:59:05.278227 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:05.278721 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:05.279728 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:05.279930 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:05.279986 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:05.282572 (+ 2586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:34451: BlockingRecv error: recv got EOF from 127.0.98.3:34451 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 03:59:05.282626 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:05.277598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45737 (local address 127.0.98.3:34451)
0926 03:59:05.277740 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 03:59:05.277758 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 03:59:05.277819 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:59:05.277955 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:59:05.277955 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:59:05.278115 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:59:05.278163 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:05.278165 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:05.278786 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:05.278787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:05.279641 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:05.279644 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:05.279995 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:05.279996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:05.280034 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:05.280493 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:59:05.280497 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:59:05.282501 (+ 2004us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45737: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":93,"threads_started":1}
Sep 26 03:59:06 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:59:06 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:59:06.050848 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:06.041771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45737 (local address 127.0.98.1:47871)
0926 03:59:06.041911 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 03:59:06.041921 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 03:59:06.041978 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:59:06.042188 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:59:06.042189 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:59:06.042402 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:59:06.042459 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:06.042461 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:06.043338 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:06.043340 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:06.043997 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:06.044000 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:06.044538 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:06.044538 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:06.044580 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:06.049037 (+ 4457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:59:06.049042 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:59:06.050788 (+ 1746us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45737: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":84,"threads_started":1}
W20260926 03:59:06.051003 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:59:06.041822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47871 (local address 127.0.98.62:45737)
0926 03:59:06.041935 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 03:59:06.041937 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:59:06.041963 (+ 26us) server_negotiation.cc:407] Connection header received
0926 03:59:06.042087 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:59:06.042088 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:59:06.042101 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:59:06.042170 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:59:06.042530 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:06.043266 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:06.044078 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:06.044462 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:06.044553 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:06.050910 (+ 6357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47871: BlockingRecv error: recv got EOF from 127.0.98.1:47871 (error 108)
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":63,"threads_started":1}
Sep 26 03:59:06 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:59:06 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:59:06.164299 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:06.159098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45737 (local address 127.0.98.2:41089)
0926 03:59:06.159283 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 03:59:06.159299 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 03:59:06.159402 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:59:06.159580 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:59:06.159581 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:59:06.159794 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:59:06.159843 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:06.159845 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:06.160610 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:06.160611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:06.161238 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:06.161240 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:06.161656 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:06.161657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:06.161735 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:06.162278 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:59:06.162282 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:59:06.164202 (+ 1920us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45737: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":97,"threads_started":1}
W20260926 03:59:06.164346 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:59:06.159193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41089 (local address 127.0.98.62:45737)
0926 03:59:06.159287 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0926 03:59:06.159290 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 03:59:06.159384 (+ 94us) server_negotiation.cc:407] Connection header received
0926 03:59:06.159492 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:59:06.159493 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:59:06.159507 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:59:06.159562 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:59:06.159951 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:06.160516 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:06.161338 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:06.161557 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:06.161655 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:06.164280 (+ 2625us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:41089: BlockingRecv error: recv got EOF from 127.0.98.2:41089 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 26 03:59:06 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:59:06 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:59:06.287964 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:06.283222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45737 (local address 127.0.98.3:37301)
0926 03:59:06.283385 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 03:59:06.283396 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 03:59:06.283447 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:59:06.283580 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:59:06.283581 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:59:06.283743 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:59:06.283791 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:06.283793 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:06.284475 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:06.284477 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:06.285096 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:06.285099 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:06.285462 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:06.285463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:06.285504 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:06.286018 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:59:06.286022 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:59:06.287907 (+ 1885us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45737: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":61,"threads_started":1}
W20260926 03:59:06.288164 9572 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:59:06.283367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37301 (local address 127.0.98.62:45737)
0926 03:59:06.283450 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0926 03:59:06.283452 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:59:06.283464 (+ 12us) server_negotiation.cc:407] Connection header received
0926 03:59:06.283493 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:59:06.283498 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:59:06.283510 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:59:06.283561 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:59:06.283860 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:06.284387 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:06.285206 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:06.285391 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:06.285450 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:06.288040 (+ 2590us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:37301: BlockingRecv error: recv got EOF from 127.0.98.3:37301 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 26 03:59:07 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:59:07 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:59:07.056041 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:07.051298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45737 (local address 127.0.98.1:41507)
0926 03:59:07.051451 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 03:59:07.051462 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 03:59:07.051528 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:59:07.051674 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:59:07.051675 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:59:07.051840 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:59:07.051889 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:07.051890 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:07.052508 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:07.052509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:07.053060 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:07.053061 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:07.053588 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:07.053589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:07.053653 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:07.054193 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:59:07.054198 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:59:07.055943 (+ 1745us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45737: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":88,"threads_started":1}
W20260926 03:59:07.056087 9576 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:59:07.051385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41507 (local address 127.0.98.62:45737)
0926 03:59:07.051476 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0926 03:59:07.051478 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:59:07.051519 (+ 41us) server_negotiation.cc:407] Connection header received
0926 03:59:07.051577 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:59:07.051578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:59:07.051587 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:59:07.051644 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:59:07.051949 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:07.052440 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:07.053128 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:07.053479 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:07.053547 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:07.056021 (+ 2474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:41507: BlockingRecv error: recv got EOF from 127.0.98.1:41507 (error 108)
Metrics: {"server-negotiator.queue_time_us":71,"thread_start_us":46,"threads_started":1}
Sep 26 03:59:07 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:59:07 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:59:07.169548 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:07.164811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45737 (local address 127.0.98.2:54523)
0926 03:59:07.164961 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 03:59:07.164973 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 03:59:07.165055 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:59:07.165232 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:59:07.165233 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:59:07.165391 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:59:07.165437 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:07.165438 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:07.166143 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:07.166144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:07.166680 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:07.166682 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:07.167084 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:07.167085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:07.167132 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:07.167685 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:59:07.167690 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:59:07.169488 (+ 1798us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45737: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":94,"threads_started":1}
W20260926 03:59:07.169687 9576 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:59:07.164880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54523 (local address 127.0.98.62:45737)
0926 03:59:07.164930 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 03:59:07.164932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:59:07.165106 (+ 174us) server_negotiation.cc:407] Connection header received
0926 03:59:07.165140 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:59:07.165140 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:59:07.165155 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:59:07.165207 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:59:07.165539 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:07.166057 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:07.166800 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:07.166989 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:07.167055 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:07.169589 (+ 2534us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54523: BlockingRecv error: recv got EOF from 127.0.98.2:54523 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 26 03:59:07 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:59:07 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:59:07.293380 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:07.288462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45737 (local address 127.0.98.3:48427)
0926 03:59:07.288617 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 03:59:07.288627 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 03:59:07.288730 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:59:07.288879 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:59:07.288879 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:59:07.289057 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:59:07.289105 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:07.289106 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:07.289721 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:07.289721 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:07.290242 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:07.290244 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:07.290560 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:07.290561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:07.290599 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:07.291105 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:59:07.291109 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:59:07.293308 (+ 2199us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45737: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":97,"threads_started":1}
W20260926 03:59:07.293561 9576 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:59:07.288564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48427 (local address 127.0.98.62:45737)
0926 03:59:07.288648 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0926 03:59:07.288650 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:59:07.288729 (+ 79us) server_negotiation.cc:407] Connection header received
0926 03:59:07.288784 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:59:07.288785 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:59:07.288796 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:59:07.288860 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:59:07.289165 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:07.289652 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:07.290310 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:07.290484 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:07.290543 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:07.293488 (+ 2945us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:48427: BlockingRecv error: recv got EOF from 127.0.98.3:48427 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 26 03:59:08 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:59:08 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:59:08.061554 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:08.056562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45737 (local address 127.0.98.1:53535)
0926 03:59:08.056765 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0926 03:59:08.056776 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 03:59:08.056841 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:59:08.056980 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:59:08.056981 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:59:08.057161 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:59:08.057209 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:08.057210 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:08.057866 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:08.057867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:08.058396 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:08.058397 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:08.058773 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:08.058773 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:08.058819 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:08.059305 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:59:08.059310 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:59:08.061453 (+ 2143us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45737: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":149,"threads_started":1}
W20260926 03:59:08.061617 9580 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:59:08.056703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53535 (local address 127.0.98.62:45737)
0926 03:59:08.056803 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0926 03:59:08.056805 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:59:08.056829 (+ 24us) server_negotiation.cc:407] Connection header received
0926 03:59:08.056901 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:59:08.056902 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:59:08.056912 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:59:08.056962 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:59:08.057286 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:08.057786 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:08.058493 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:08.058694 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:08.058761 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:08.061544 (+ 2783us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:53535: BlockingRecv error: recv got EOF from 127.0.98.1:53535 (error 108)
Metrics: {"server-negotiator.queue_time_us":79,"thread_start_us":49,"threads_started":1}
Sep 26 03:59:08 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:59:08 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:59:08.174621 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:08.170055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45737 (local address 127.0.98.2:35949)
0926 03:59:08.170199 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 03:59:08.170209 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 03:59:08.170262 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:59:08.170392 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:59:08.170393 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:59:08.170553 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:59:08.170601 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:08.170602 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:08.171258 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:08.171259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:08.171809 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:08.171810 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:08.172185 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:08.172186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:08.172239 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:08.172733 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:59:08.172738 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:59:08.174563 (+ 1825us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45737: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":93,"threads_started":1}
W20260926 03:59:08.174762 9580 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:59:08.170132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35949 (local address 127.0.98.62:45737)
0926 03:59:08.170216 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0926 03:59:08.170218 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:59:08.170251 (+ 33us) server_negotiation.cc:407] Connection header received
0926 03:59:08.170308 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:59:08.170308 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:59:08.170319 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:59:08.170374 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:59:08.170664 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:08.171170 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:08.171902 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:08.172094 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:08.172157 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:08.174681 (+ 2524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35949: BlockingRecv error: recv got EOF from 127.0.98.2:35949 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 26 03:59:08 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:59:08 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:59:08.298648 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:08.293917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45737 (local address 127.0.98.3:50759)
0926 03:59:08.294067 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 03:59:08.294078 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 03:59:08.294129 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:59:08.294312 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:59:08.294313 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:59:08.294473 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:59:08.294521 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:08.294523 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:08.295156 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:08.295158 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:08.295699 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:08.295700 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:08.296014 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:08.296015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:08.296054 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:08.296526 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:59:08.296530 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:59:08.298536 (+ 2006us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45737: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":85,"threads_started":1}
W20260926 03:59:08.298728 9580 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:59:08.293984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50759 (local address 127.0.98.62:45737)
0926 03:59:08.294156 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 03:59:08.294158 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:59:08.294168 (+ 10us) server_negotiation.cc:407] Connection header received
0926 03:59:08.294192 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:59:08.294193 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:59:08.294205 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:59:08.294291 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:59:08.294596 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:08.295082 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:08.295775 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:08.295953 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:08.296003 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:08.298650 (+ 2647us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50759: BlockingRecv error: recv got EOF from 127.0.98.3:50759 (error 108)
Metrics: {"server-negotiator.queue_time_us":142}
Sep 26 03:59:09 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:59:09 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:59:09.067085 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:09.062051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45737 (local address 127.0.98.1:42007)
0926 03:59:09.062222 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 03:59:09.062248 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 03:59:09.062315 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:59:09.062650 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:59:09.062651 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:59:09.062824 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:59:09.062874 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:09.062876 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:09.063571 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:09.063572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:09.064114 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:09.064115 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:09.064526 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:09.064527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:09.064571 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:09.065152 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:59:09.065157 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:59:09.067026 (+ 1869us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45737: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":114,"threads_started":1}
W20260926 03:59:09.067250 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:59:09.062123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42007 (local address 127.0.98.62:45737)
0926 03:59:09.062234 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 03:59:09.062236 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:59:09.062314 (+ 78us) server_negotiation.cc:407] Connection header received
0926 03:59:09.062360 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:59:09.062361 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:59:09.062374 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:59:09.062627 (+ 253us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:59:09.062954 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:09.063476 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:09.064244 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:09.064441 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:09.064508 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:09.067148 (+ 2640us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42007: BlockingRecv error: recv got EOF from 127.0.98.1:42007 (error 108)
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":63,"threads_started":1}
Sep 26 03:59:09 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:59:09 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:59:09.180013 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:09.175097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45737 (local address 127.0.98.2:59331)
0926 03:59:09.175251 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 03:59:09.175264 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 03:59:09.175379 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:59:09.175553 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:59:09.175554 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:59:09.175739 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:59:09.175785 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:09.175787 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:09.176523 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:09.176525 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:09.177093 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:09.177095 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:09.177493 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:09.177494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:09.177559 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:09.178048 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:59:09.178052 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:59:09.179910 (+ 1858us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45737: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":57,"threads_started":1}
W20260926 03:59:09.180068 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:59:09.175279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59331 (local address 127.0.98.62:45737)
0926 03:59:09.175346 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 03:59:09.175348 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:59:09.175358 (+ 10us) server_negotiation.cc:407] Connection header received
0926 03:59:09.175456 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:59:09.175457 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:59:09.175468 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:59:09.175543 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:59:09.175872 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:09.176430 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:09.177185 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:09.177387 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:09.177479 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:09.180003 (+ 2524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:59331: BlockingRecv error: recv got EOF from 127.0.98.2:59331 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 26 03:59:09 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:59:09 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:59:09.303911 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:09.299179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45737 (local address 127.0.98.3:46805)
0926 03:59:09.299330 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 03:59:09.299341 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 03:59:09.299423 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:59:09.299588 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:59:09.299589 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:59:09.299748 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:59:09.299824 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:09.299826 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:09.300524 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:09.300524 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:09.301180 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:09.301183 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:09.301508 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:09.301509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:09.301547 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:09.302033 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:59:09.302038 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:59:09.303843 (+ 1805us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45737: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":90,"threads_started":1}
W20260926 03:59:09.304001 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:59:09.299243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46805 (local address 127.0.98.62:45737)
0926 03:59:09.299312 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 03:59:09.299315 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 03:59:09.299421 (+ 106us) server_negotiation.cc:407] Connection header received
0926 03:59:09.299482 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:59:09.299483 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:59:09.299512 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:59:09.299553 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:59:09.299904 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:09.300457 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:09.301252 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:09.301446 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:09.301499 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:09.303934 (+ 2435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46805: BlockingRecv error: recv got EOF from 127.0.98.3:46805 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 26 03:59:10 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:59:10 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:59:10.072009 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:10.067586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45737 (local address 127.0.98.1:45187)
0926 03:59:10.067739 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 03:59:10.067751 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 03:59:10.067825 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:59:10.067998 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:59:10.067999 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:59:10.068188 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:59:10.068236 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:10.068237 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:10.068921 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:10.068922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:10.069443 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:10.069445 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:10.069811 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:10.069812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:10.069855 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:10.070299 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:59:10.070303 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:59:10.071950 (+ 1647us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45737: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":86,"threads_started":1}
W20260926 03:59:10.072131 9588 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:59:10.067673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45187 (local address 127.0.98.62:45737)
0926 03:59:10.067787 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 03:59:10.067790 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 03:59:10.067806 (+ 16us) server_negotiation.cc:407] Connection header received
0926 03:59:10.067898 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:59:10.067899 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:59:10.067912 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:59:10.067978 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:59:10.068327 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:10.068848 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:10.069529 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:10.069727 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:10.069785 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:10.072034 (+ 2249us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45187: BlockingRecv error: recv got EOF from 127.0.98.1:45187 (error 108)
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":45,"threads_started":1}
Sep 26 03:59:10 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:59:10 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:59:10.185433 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:10.180556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45737 (local address 127.0.98.2:60547)
0926 03:59:10.180755 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 03:59:10.180769 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 03:59:10.180870 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:59:10.181047 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:59:10.181048 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:59:10.181229 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:59:10.181299 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:10.181301 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:10.181980 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:10.181982 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:10.182521 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:10.182522 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:10.182902 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:10.182903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:10.182957 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:10.183410 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:59:10.183414 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:59:10.185325 (+ 1911us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45737: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":109,"threads_started":1}
W20260926 03:59:10.185469 9588 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:59:10.180685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60547 (local address 127.0.98.62:45737)
0926 03:59:10.180759 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 03:59:10.180762 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 03:59:10.180847 (+ 85us) server_negotiation.cc:407] Connection header received
0926 03:59:10.180949 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:59:10.180950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:59:10.180963 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:59:10.181041 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:59:10.181388 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:10.181899 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:10.182621 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:10.182829 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:10.182904 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:10.185403 (+ 2499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:60547: BlockingRecv error: recv got EOF from 127.0.98.2:60547 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 26 03:59:10 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:59:10 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:59:10.308851 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:10.304440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45737 (local address 127.0.98.3:49875)
0926 03:59:10.304569 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 03:59:10.304579 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 03:59:10.304693 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:59:10.304855 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:59:10.304856 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:59:10.305020 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:59:10.305069 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:10.305071 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:10.305691 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:10.305691 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:10.306210 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:10.306212 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:10.306522 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:10.306523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:10.306562 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:10.307004 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:59:10.307008 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:59:10.308790 (+ 1782us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45737: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":63,"threads_started":1}
W20260926 03:59:10.308924 9588 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:59:10.304483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49875 (local address 127.0.98.62:45737)
0926 03:59:10.304537 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 03:59:10.304539 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:59:10.304667 (+ 128us) server_negotiation.cc:407] Connection header received
0926 03:59:10.304774 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:59:10.304774 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:59:10.304786 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:59:10.304837 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:59:10.305137 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:10.305624 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:10.306280 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:10.306461 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:10.306511 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:10.308875 (+ 2364us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:49875: BlockingRecv error: recv got EOF from 127.0.98.3:49875 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Sep 26 03:59:11 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:59:11 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:59:11.081257 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:11.072607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45737 (local address 127.0.98.1:38489)
0926 03:59:11.072795 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 03:59:11.072821 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 03:59:11.072871 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:59:11.073083 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:59:11.073085 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:59:11.073304 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:59:11.073354 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:11.073356 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:11.074304 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:11.074305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:11.074834 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:11.074836 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:11.075365 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:11.075367 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:11.075436 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:11.079239 (+ 3803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:59:11.079252 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:59:11.081155 (+ 1903us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45737: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":132,"threads_started":1}
W20260926 03:59:11.081347 9592 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:59:11.072764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38489 (local address 127.0.98.62:45737)
0926 03:59:11.072901 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 03:59:11.072903 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:59:11.072914 (+ 11us) server_negotiation.cc:407] Connection header received
0926 03:59:11.072948 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:59:11.072948 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:59:11.072958 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:59:11.073030 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:59:11.073463 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:11.074208 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:11.074985 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:11.075279 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:11.075361 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:11.081275 (+ 5914us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38489: BlockingRecv error: recv got EOF from 127.0.98.1:38489 (error 108)
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":92,"threads_started":1}
Sep 26 03:59:11 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 03:59:11 dist-test-slave-nhpw krb5kdc[8532](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 03:59:11.190702 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:11.186078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45737 (local address 127.0.98.2:39837)
0926 03:59:11.186248 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 03:59:11.186260 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 03:59:11.186355 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:59:11.186514 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:59:11.186515 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:59:11.186695 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:59:11.186741 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:11.186742 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:11.187421 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:11.187422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:11.187961 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:11.187962 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:11.188330 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:11.188331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:11.188387 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:11.188869 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:59:11.188873 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:59:11.190604 (+ 1731us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:45737: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":71,"threads_started":1}
W20260926 03:59:11.190749 9592 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:59:11.186139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39837 (local address 127.0.98.62:45737)
0926 03:59:11.186212 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 03:59:11.186215 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 03:59:11.186337 (+ 122us) server_negotiation.cc:407] Connection header received
0926 03:59:11.186417 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:59:11.186417 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:59:11.186433 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:59:11.186509 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:59:11.186827 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:11.187333 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:11.188050 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:11.188246 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:11.188332 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:11.190685 (+ 2353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:39837: BlockingRecv error: recv got EOF from 127.0.98.2:39837 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 03:59:11.227761 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 9115
I20260926 03:59:11.232420 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 9267
I20260926 03:59:11.236848 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 9400
I20260926 03:59:11.241050 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 9045
2026-09-26T03:59:11Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17545 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-26T03:59:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:59:11Z Disabled control of system clock
I20260926 03:59:11.269333 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395095474643-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395095474643-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395095474643-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46129
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46129
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 03:59:11.351183 9597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:59:11.351411 9597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:59:11.351432 9597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:59:11.352596 9597 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:59:11.352664 9597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:59:11.352679 9597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:59:11.352690 9597 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:59:11.352702 9597 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:59:11.354030 9597 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395095474643-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395095474643-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46129
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9597
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395095474643-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:59:11.354238 9597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:59:11.354481 9597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:59:11.359932 9602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:59:11.359941 9605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:59:11.359980 9603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:59:11.359983 9597 server_base.cc:1034] running on GCE node
I20260926 03:59:11.360529 9597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:59:11.360939 9597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:59:11.362115 9597 hybrid_clock.cc:648] HybridClock initialized: now 1790395151362097 us; error 38 us; skew 500 ppm
I20260926 03:59:11.363757 9597 webserver.cc:460] Webserver started at http://127.0.98.62:40417/ using document root <none> and password file <none>
I20260926 03:59:11.363979 9597 fs_manager.cc:359] Metadata directory not provided
I20260926 03:59:11.364023 9597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:59:11.364115 9597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:59:11.365264 9597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395095474643-392-0/minicluster-data/master-0/data/instance:
uuid: "b69e7fad05ce497b81e2e5c86e6c8886"
format_stamp: "Formatted at 2026-09-26 03:59:11 on dist-test-slave-nhpw"
server_key: "3c6a8cd76577799d5fd52f0ef740ed6d"
server_key_iv: "2152a2f4befb99bdf0752fef99d3f1f4"
server_key_version: "encryptionkey@0"
I20260926 03:59:11.365590 9597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395095474643-392-0/minicluster-data/master-0/wal/instance:
uuid: "b69e7fad05ce497b81e2e5c86e6c8886"
format_stamp: "Formatted at 2026-09-26 03:59:11 on dist-test-slave-nhpw"
server_key: "3c6a8cd76577799d5fd52f0ef740ed6d"
server_key_iv: "2152a2f4befb99bdf0752fef99d3f1f4"
server_key_version: "encryptionkey@0"
I20260926 03:59:11.366889 9597 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 03:59:11.367724 9611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:59:11.367970 9597 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 03:59:11.368152 9597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395095474643-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395095474643-392-0/minicluster-data/master-0/wal
uuid: "b69e7fad05ce497b81e2e5c86e6c8886"
format_stamp: "Formatted at 2026-09-26 03:59:11 on dist-test-slave-nhpw"
server_key: "3c6a8cd76577799d5fd52f0ef740ed6d"
server_key_iv: "2152a2f4befb99bdf0752fef99d3f1f4"
server_key_version: "encryptionkey@0"
I20260926 03:59:11.368283 9597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395095474643-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395095474643-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395095474643-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:59:11.393021 9597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:59:11.393359 9597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:59:11.393474 9597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:59:11.397619 9597 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46129
I20260926 03:59:11.397658 9663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46129 every 8 connection(s)
I20260926 03:59:11.398018 9597 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
I20260926 03:59:11.398707 9664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:59:11.402045 9664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b69e7fad05ce497b81e2e5c86e6c8886: Bootstrap starting.
I20260926 03:59:11.402618 9664 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b69e7fad05ce497b81e2e5c86e6c8886: Neither blocks nor log segments found. Creating new log.
I20260926 03:59:11.402860 9664 log.cc:824] T 00000000000000000000000000000000 P b69e7fad05ce497b81e2e5c86e6c8886: Log is configured to *not* fsync() on all Append() calls
I20260926 03:59:11.403580 9664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b69e7fad05ce497b81e2e5c86e6c8886: No bootstrap required, opened a new log
I20260926 03:59:11.404096 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 9597
I20260926 03:59:11.404201 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395095474643-392-0/minicluster-data/master-0/wal/instance
I20260926 03:59:11.404405 392 external_mini_cluster.cc:1428] Setting key 1640a6fd4f5d53b775ff0524dd6ac747
I20260926 03:59:11.405642 9664 raft_consensus.cc:348] T 00000000000000000000000000000000 P b69e7fad05ce497b81e2e5c86e6c8886 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b69e7fad05ce497b81e2e5c86e6c8886" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46129 } }
I20260926 03:59:11.405815 9664 raft_consensus.cc:374] T 00000000000000000000000000000000 P b69e7fad05ce497b81e2e5c86e6c8886 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:59:11.405854 9664 raft_consensus.cc:729] T 00000000000000000000000000000000 P b69e7fad05ce497b81e2e5c86e6c8886 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b69e7fad05ce497b81e2e5c86e6c8886, State: Initialized, Role: FOLLOWER
I20260926 03:59:11.406029 9664 consensus_queue.cc:260] T 00000000000000000000000000000000 P b69e7fad05ce497b81e2e5c86e6c8886 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b69e7fad05ce497b81e2e5c86e6c8886" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46129 } }
I20260926 03:59:11.406128 9664 raft_consensus.cc:388] T 00000000000000000000000000000000 P b69e7fad05ce497b81e2e5c86e6c8886 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:59:11.406193 9664 raft_consensus.cc:482] T 00000000000000000000000000000000 P b69e7fad05ce497b81e2e5c86e6c8886 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:59:11.406242 9664 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b69e7fad05ce497b81e2e5c86e6c8886 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:59:11.406795 9664 raft_consensus.cc:504] T 00000000000000000000000000000000 P b69e7fad05ce497b81e2e5c86e6c8886 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b69e7fad05ce497b81e2e5c86e6c8886" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46129 } }
I20260926 03:59:11.406881 9664 leader_election.cc:302] T 00000000000000000000000000000000 P b69e7fad05ce497b81e2e5c86e6c8886 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b69e7fad05ce497b81e2e5c86e6c8886; no voters:
I20260926 03:59:11.407080 9664 leader_election.cc:288] T 00000000000000000000000000000000 P b69e7fad05ce497b81e2e5c86e6c8886 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:59:11.407112 9669 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b69e7fad05ce497b81e2e5c86e6c8886 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:59:11.407217 9669 raft_consensus.cc:686] T 00000000000000000000000000000000 P b69e7fad05ce497b81e2e5c86e6c8886 [term 1 LEADER]: Becoming Leader. State: Replica: b69e7fad05ce497b81e2e5c86e6c8886, State: Running, Role: LEADER
I20260926 03:59:11.407356 9669 consensus_queue.cc:237] T 00000000000000000000000000000000 P b69e7fad05ce497b81e2e5c86e6c8886 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b69e7fad05ce497b81e2e5c86e6c8886" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46129 } }
I20260926 03:59:11.407358 9664 sys_catalog.cc:564] T 00000000000000000000000000000000 P b69e7fad05ce497b81e2e5c86e6c8886 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:59:11.407907 9670 sys_catalog.cc:455] T 00000000000000000000000000000000 P b69e7fad05ce497b81e2e5c86e6c8886 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b69e7fad05ce497b81e2e5c86e6c8886" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b69e7fad05ce497b81e2e5c86e6c8886" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46129 } } }
I20260926 03:59:11.407994 9670 sys_catalog.cc:458] T 00000000000000000000000000000000 P b69e7fad05ce497b81e2e5c86e6c8886 [sys.catalog]: This master's current role is: LEADER
I20260926 03:59:11.408129 9671 sys_catalog.cc:455] T 00000000000000000000000000000000 P b69e7fad05ce497b81e2e5c86e6c8886 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b69e7fad05ce497b81e2e5c86e6c8886. Latest consensus state: current_term: 1 leader_uuid: "b69e7fad05ce497b81e2e5c86e6c8886" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b69e7fad05ce497b81e2e5c86e6c8886" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46129 } } }
I20260926 03:59:11.408248 9671 sys_catalog.cc:458] T 00000000000000000000000000000000 P b69e7fad05ce497b81e2e5c86e6c8886 [sys.catalog]: This master's current role is: LEADER
I20260926 03:59:11.408550 9676 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:59:11.409123 9676 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:59:11.410786 9676 catalog_manager.cc:1347] Generated new cluster ID: d877620673ca44e3bc596cdd440cd511
I20260926 03:59:11.410861 9676 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:59:11.411298 9668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:11.405001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45544 (local address 127.0.98.62:46129)
0926 03:59:11.405220 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 03:59:11.405228 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 03:59:11.405257 (+ 29us) server_negotiation.cc:407] Connection header received
0926 03:59:11.405616 (+ 359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:59:11.405623 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:59:11.405692 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:59:11.405848 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:59:11.406087 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:11.407665 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:11.408858 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:11.409481 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:11.409606 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:11.410489 (+ 883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:59:11.410493 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:59:11.410493 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:59:11.410698 (+ 205us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:59:11.410745 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:59:11.410758 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:59:11.410826 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 03:59:11.411031 (+ 205us) server_negotiation.cc:299] Negotiation successful
0926 03:59:11.411106 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":94,"threads_started":1}
I20260926 03:59:11.416025 9676 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:59:11.416558 9676 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:59:11.421018 9676 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b69e7fad05ce497b81e2e5c86e6c8886: Generated new TSK 0
I20260926 03:59:11.421300 9676 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:59:11.427650 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395095474643-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395095474643-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395095474643-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46129
--builtin_ntp_servers=127.0.98.20:34593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 03:59:11.519038 9688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:59:11.519259 9688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:59:11.519281 9688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:59:11.520517 9688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:59:11.520565 9688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:59:11.520591 9688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 03:59:11.522007 9688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395095474643-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395095474643-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9688
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395095474643-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:59:11.522262 9688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:59:11.522593 9688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:59:11.528560 9693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:59:11.528555 9696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:59:11.528555 9694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:59:11.528918 9688 server_base.cc:1034] running on GCE node
I20260926 03:59:11.529167 9688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:59:11.529422 9688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:59:11.530615 9688 hybrid_clock.cc:648] HybridClock initialized: now 1790395151530592 us; error 42 us; skew 500 ppm
I20260926 03:59:11.532011 9688 webserver.cc:460] Webserver started at http://127.0.98.1:34091/ using document root <none> and password file <none>
I20260926 03:59:11.532271 9688 fs_manager.cc:359] Metadata directory not provided
I20260926 03:59:11.532343 9688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:59:11.532490 9688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:59:11.533659 9688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395095474643-392-0/minicluster-data/ts-0/data/instance:
uuid: "05303e4e81dd4ad19ed049225365ef49"
format_stamp: "Formatted at 2026-09-26 03:59:11 on dist-test-slave-nhpw"
server_key: "ce8d88dc512482d6831bb405fa8f1101"
server_key_iv: "f89e8625ed0559540de84d56dd3c3181"
server_key_version: "encryptionkey@0"
I20260926 03:59:11.534042 9688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395095474643-392-0/minicluster-data/ts-0/wal/instance:
uuid: "05303e4e81dd4ad19ed049225365ef49"
format_stamp: "Formatted at 2026-09-26 03:59:11 on dist-test-slave-nhpw"
server_key: "ce8d88dc512482d6831bb405fa8f1101"
server_key_iv: "f89e8625ed0559540de84d56dd3c3181"
server_key_version: "encryptionkey@0"
I20260926 03:59:11.535442 9688 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 03:59:11.536231 9702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:59:11.536404 9688 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 03:59:11.536497 9688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395095474643-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395095474643-392-0/minicluster-data/ts-0/wal
uuid: "05303e4e81dd4ad19ed049225365ef49"
format_stamp: "Formatted at 2026-09-26 03:59:11 on dist-test-slave-nhpw"
server_key: "ce8d88dc512482d6831bb405fa8f1101"
server_key_iv: "f89e8625ed0559540de84d56dd3c3181"
server_key_version: "encryptionkey@0"
I20260926 03:59:11.536583 9688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395095474643-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395095474643-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395095474643-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:59:11.565713 9688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:59:11.566057 9688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:59:11.566222 9688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:59:11.566466 9688 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:59:11.566811 9688 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:59:11.566876 9688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:59:11.566941 9688 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:59:11.566972 9688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:59:11.573254 9688 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44175
I20260926 03:59:11.573330 9815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44175 every 8 connection(s)
I20260926 03:59:11.573688 9688 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
I20260926 03:59:11.579005 9668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:11.574440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49727 (local address 127.0.98.62:46129)
0926 03:59:11.574490 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 03:59:11.574492 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:59:11.574700 (+ 208us) server_negotiation.cc:407] Connection header received
0926 03:59:11.575252 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:59:11.575253 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:59:11.575269 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:59:11.575333 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:59:11.576009 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:11.576524 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:11.577577 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:11.577775 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:11.577866 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:11.578699 (+ 833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:59:11.578700 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:59:11.578700 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:59:11.578707 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:59:11.578722 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:59:11.578727 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:59:11.578796 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 03:59:11.578898 (+ 102us) server_negotiation.cc:299] Negotiation successful
0926 03:59:11.578926 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 03:59:11.579226 9818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:11.574396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46129 (local address 127.0.98.1:49727)
0926 03:59:11.574567 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 03:59:11.574607 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 03:59:11.575137 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:59:11.575413 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:59:11.575417 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:59:11.575427 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:59:11.575910 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:11.575914 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:11.576619 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:11.576621 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:11.577465 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:11.577467 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:11.577885 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:11.577886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:11.578025 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:11.578549 (+ 524us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:59:11.578550 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:59:11.578578 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:59:11.578600 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:59:11.578794 (+ 194us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:59:11.578800 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:59:11.578803 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 03:59:11.578895 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 03:59:11.578998 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":68,"threads_started":1}
I20260926 03:59:11.579734 9816 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46129
I20260926 03:59:11.579835 9816 heartbeater.cc:464] Registering TS with master...
I20260926 03:59:11.580032 9816 heartbeater.cc:511] Master 127.0.98.62:46129 requested a full tablet report, sending...
I20260926 03:59:11.580569 9628 ts_manager.cc:194] Registered new tserver with Master: 05303e4e81dd4ad19ed049225365ef49 (127.0.98.1:44175)
I20260926 03:59:11.581571 9628 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:49727
I20260926 03:59:11.583777 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 9688
I20260926 03:59:11.583889 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395095474643-392-0/minicluster-data/ts-0/wal/instance
I20260926 03:59:11.584075 392 external_mini_cluster.cc:1428] Setting key e4a7a2f67b0ea8fca9319e2fd0a53b2b
I20260926 03:59:11.584970 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395095474643-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395095474643-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395095474643-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46129
--builtin_ntp_servers=127.0.98.20:34593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 03:59:11.667043 9819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:59:11.667245 9819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:59:11.667266 9819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:59:11.668372 9819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:59:11.668407 9819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:59:11.668431 9819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 03:59:11.669816 9819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395095474643-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395095474643-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395095474643-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:59:11.670017 9819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:59:11.670251 9819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:59:11.675446 9825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:59:11.675454 9824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:59:11.675454 9827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:59:11.675768 9819 server_base.cc:1034] running on GCE node
I20260926 03:59:11.675968 9819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:59:11.676239 9819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:59:11.677414 9819 hybrid_clock.cc:648] HybridClock initialized: now 1790395151677404 us; error 29 us; skew 500 ppm
I20260926 03:59:11.678748 9819 webserver.cc:460] Webserver started at http://127.0.98.2:40513/ using document root <none> and password file <none>
I20260926 03:59:11.678996 9819 fs_manager.cc:359] Metadata directory not provided
I20260926 03:59:11.679066 9819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:59:11.679232 9819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:59:11.680330 9819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395095474643-392-0/minicluster-data/ts-1/data/instance:
uuid: "73129e9e7ef740d78767cc92682f7523"
format_stamp: "Formatted at 2026-09-26 03:59:11 on dist-test-slave-nhpw"
server_key: "b5050b3c1160307124e93f10f68040dd"
server_key_iv: "295f0889d7cd4d3605ce452c5e86bdb9"
server_key_version: "encryptionkey@0"
I20260926 03:59:11.680617 9819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395095474643-392-0/minicluster-data/ts-1/wal/instance:
uuid: "73129e9e7ef740d78767cc92682f7523"
format_stamp: "Formatted at 2026-09-26 03:59:11 on dist-test-slave-nhpw"
server_key: "b5050b3c1160307124e93f10f68040dd"
server_key_iv: "295f0889d7cd4d3605ce452c5e86bdb9"
server_key_version: "encryptionkey@0"
I20260926 03:59:11.681995 9819 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 03:59:11.682822 9833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:59:11.683154 9819 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 03:59:11.683297 9819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395095474643-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395095474643-392-0/minicluster-data/ts-1/wal
uuid: "73129e9e7ef740d78767cc92682f7523"
format_stamp: "Formatted at 2026-09-26 03:59:11 on dist-test-slave-nhpw"
server_key: "b5050b3c1160307124e93f10f68040dd"
server_key_iv: "295f0889d7cd4d3605ce452c5e86bdb9"
server_key_version: "encryptionkey@0"
I20260926 03:59:11.683378 9819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395095474643-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395095474643-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395095474643-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:59:11.699836 9819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:59:11.700137 9819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:59:11.700320 9819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:59:11.700551 9819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:59:11.700898 9819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:59:11.700955 9819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:59:11.700997 9819 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:59:11.701025 9819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:59:11.706782 9819 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35211
I20260926 03:59:11.706823 9946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35211 every 8 connection(s)
I20260926 03:59:11.707144 9819 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
I20260926 03:59:11.709792 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 9819
I20260926 03:59:11.709906 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395095474643-392-0/minicluster-data/ts-1/wal/instance
I20260926 03:59:11.710084 392 external_mini_cluster.cc:1428] Setting key 9f2f21163b4a1a5b0ec3153adcaa6af7
I20260926 03:59:11.711176 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395095474643-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395095474643-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395095474643-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46129
--builtin_ntp_servers=127.0.98.20:34593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 03:59:11.713352 9668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:11.707858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60781 (local address 127.0.98.62:46129)
0926 03:59:11.707901 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 03:59:11.707903 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:59:11.708170 (+ 267us) server_negotiation.cc:407] Connection header received
0926 03:59:11.708831 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:59:11.708833 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:59:11.708848 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:59:11.708923 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:59:11.709653 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:11.710471 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:11.711715 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:11.711920 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:11.712008 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:11.713013 (+ 1005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:59:11.713015 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:59:11.713016 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:59:11.713021 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:59:11.713041 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:59:11.713046 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:59:11.713107 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 03:59:11.713223 (+ 116us) server_negotiation.cc:299] Negotiation successful
0926 03:59:11.713253 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 03:59:11.713590 9949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:11.707845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46129 (local address 127.0.98.2:60781)
0926 03:59:11.708045 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 03:59:11.708078 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 03:59:11.708715 (+ 637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:59:11.709016 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:59:11.709019 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:59:11.709052 (+ 33us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:59:11.709544 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:11.709548 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:11.710581 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:11.710582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:11.711608 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:11.711611 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:11.712019 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:11.712020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:11.712153 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:11.712847 (+ 694us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:59:11.712848 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:59:11.712887 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:59:11.712915 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:59:11.713122 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:59:11.713129 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:59:11.713133 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 03:59:11.713218 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 03:59:11.713353 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":88,"threads_started":1}
I20260926 03:59:11.714354 9947 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46129
I20260926 03:59:11.714448 9947 heartbeater.cc:464] Registering TS with master...
I20260926 03:59:11.714654 9947 heartbeater.cc:511] Master 127.0.98.62:46129 requested a full tablet report, sending...
I20260926 03:59:11.715034 9628 ts_manager.cc:194] Registered new tserver with Master: 73129e9e7ef740d78767cc92682f7523 (127.0.98.2:35211)
I20260926 03:59:11.715425 9628 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:60781
W20260926 03:59:11.793462 9950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:59:11.793706 9950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:59:11.793740 9950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:59:11.794883 9950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:59:11.794948 9950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:59:11.794991 9950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 03:59:11.796396 9950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395095474643-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395095474643-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9950
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395095474643-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:59:11.796718 9950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:59:11.797024 9950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:59:11.802435 9958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:59:11.802570 9950 server_base.cc:1034] running on GCE node
W20260926 03:59:11.802377 9956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:59:11.802377 9955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:59:11.802923 9950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:59:11.803201 9950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:59:11.804379 9950 hybrid_clock.cc:648] HybridClock initialized: now 1790395151804364 us; error 28 us; skew 500 ppm
I20260926 03:59:11.805749 9950 webserver.cc:460] Webserver started at http://127.0.98.3:36171/ using document root <none> and password file <none>
I20260926 03:59:11.806025 9950 fs_manager.cc:359] Metadata directory not provided
I20260926 03:59:11.806097 9950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:59:11.806242 9950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:59:11.807410 9950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395095474643-392-0/minicluster-data/ts-2/data/instance:
uuid: "322ab11942cf4d7faa1192f57db0282f"
format_stamp: "Formatted at 2026-09-26 03:59:11 on dist-test-slave-nhpw"
server_key: "5589045ca81e3318c0f8d711c9d2fb59"
server_key_iv: "6bc93a0faf7a3d980ca4d08acf3322eb"
server_key_version: "encryptionkey@0"
I20260926 03:59:11.807808 9950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395095474643-392-0/minicluster-data/ts-2/wal/instance:
uuid: "322ab11942cf4d7faa1192f57db0282f"
format_stamp: "Formatted at 2026-09-26 03:59:11 on dist-test-slave-nhpw"
server_key: "5589045ca81e3318c0f8d711c9d2fb59"
server_key_iv: "6bc93a0faf7a3d980ca4d08acf3322eb"
server_key_version: "encryptionkey@0"
I20260926 03:59:11.809279 9950 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 03:59:11.810014 9964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:59:11.810236 9950 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 03:59:11.810339 9950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395095474643-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395095474643-392-0/minicluster-data/ts-2/wal
uuid: "322ab11942cf4d7faa1192f57db0282f"
format_stamp: "Formatted at 2026-09-26 03:59:11 on dist-test-slave-nhpw"
server_key: "5589045ca81e3318c0f8d711c9d2fb59"
server_key_iv: "6bc93a0faf7a3d980ca4d08acf3322eb"
server_key_version: "encryptionkey@0"
I20260926 03:59:11.810412 9950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395095474643-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395095474643-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395095474643-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:59:11.825825 9950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:59:11.826148 9950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:59:11.826298 9950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:59:11.826542 9950 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:59:11.826872 9950 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:59:11.826906 9950 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:59:11.826929 9950 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:59:11.826974 9950 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:59:11.832571 9950 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46695
I20260926 03:59:11.832671 10077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46695 every 8 connection(s)
I20260926 03:59:11.833014 9950 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
I20260926 03:59:11.836793 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 9950
I20260926 03:59:11.836877 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790395095474643-392-0/minicluster-data/ts-2/wal/instance
I20260926 03:59:11.837049 392 external_mini_cluster.cc:1428] Setting key 7fa32e7682341932ead2fd3be3f8d173
I20260926 03:59:11.838740 9668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:11.833775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45969 (local address 127.0.98.62:46129)
0926 03:59:11.833857 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0926 03:59:11.833860 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 03:59:11.834111 (+ 251us) server_negotiation.cc:407] Connection header received
0926 03:59:11.834589 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:59:11.834590 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:59:11.834600 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:59:11.834662 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:59:11.835365 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:11.835857 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:11.836966 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:11.837263 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:11.837332 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:11.838451 (+ 1119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:59:11.838453 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:59:11.838453 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:59:11.838457 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:59:11.838486 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:59:11.838490 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:59:11.838565 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 03:59:11.838648 (+ 83us) server_negotiation.cc:299] Negotiation successful
0926 03:59:11.838683 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 03:59:11.838955 10080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:11.833750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46129 (local address 127.0.98.3:45969)
0926 03:59:11.833966 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0926 03:59:11.833999 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 03:59:11.834485 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:59:11.834742 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:59:11.834746 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:59:11.834754 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:59:11.835241 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:11.835245 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:11.835936 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:11.835937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:11.836869 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:11.836873 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:11.837418 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:11.837419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:11.837615 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:11.838231 (+ 616us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:59:11.838231 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:59:11.838302 (+ 71us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:59:11.838333 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:59:11.838574 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:59:11.838579 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:59:11.838584 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 03:59:11.838648 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 03:59:11.838735 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":111,"threads_started":1}
I20260926 03:59:11.839372 10078 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46129
I20260926 03:59:11.839457 10078 heartbeater.cc:464] Registering TS with master...
I20260926 03:59:11.839599 10078 heartbeater.cc:511] Master 127.0.98.62:46129 requested a full tablet report, sending...
I20260926 03:59:11.839983 9628 ts_manager.cc:194] Registered new tserver with Master: 322ab11942cf4d7faa1192f57db0282f (127.0.98.3:46695)
I20260926 03:59:11.840317 9628 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:45969
I20260926 03:59:11.849229 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 03:59:11.851162 9668 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:59:11.850608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45558 (local address 127.0.98.62:46129)
0926 03:59:11.850677 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 03:59:11.850679 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:59:11.850825 (+ 146us) server_negotiation.cc:407] Connection header received
0926 03:59:11.850862 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:59:11.850863 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:59:11.850875 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:59:11.850930 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:59:11.851073 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45558: BlockingRecv error: recv got EOF from 127.0.0.1:45558 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 03:59:11.851174 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 9688
I20260926 03:59:11.855844 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 9819
I20260926 03:59:11.860049 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 9950
I20260926 03:59:11.864297 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 9597
2026-09-26T03:59:11Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (621 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-26T03:59:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:59:11Z Disabled control of system clock
I20260926 03:59:11.891400 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395095474643-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395095474643-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395095474643-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42213
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42213
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 03:59:11.973707 10089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:59:11.973943 10089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:59:11.973966 10089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:59:11.975184 10089 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:59:11.975234 10089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:59:11.975247 10089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:59:11.975260 10089 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:59:11.975272 10089 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:59:11.976935 10089 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395095474643-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395095474643-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42213
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10089
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395095474643-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:59:11.977224 10089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:59:11.977541 10089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:59:11.982345 10095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:59:11.982409 10097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:59:11.982403 10094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:59:11.982625 10089 server_base.cc:1034] running on GCE node
I20260926 03:59:11.982941 10089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:59:11.983276 10089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:59:11.984470 10089 hybrid_clock.cc:648] HybridClock initialized: now 1790395151984453 us; error 44 us; skew 500 ppm
I20260926 03:59:11.985827 10089 webserver.cc:460] Webserver started at http://127.0.98.62:45693/ using document root <none> and password file <none>
I20260926 03:59:11.986186 10089 fs_manager.cc:359] Metadata directory not provided
I20260926 03:59:11.986259 10089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:59:11.986413 10089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:59:11.987493 10089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395095474643-392-0/minicluster-data/master-0/data/instance:
uuid: "464529f3459d46438c1d94067da12d1c"
format_stamp: "Formatted at 2026-09-26 03:59:11 on dist-test-slave-nhpw"
server_key: "97293130fc04ec9d8cbbac92f0cd0087"
server_key_iv: "03e23e446881744c79896a4e1a67ad8f"
server_key_version: "encryptionkey@0"
I20260926 03:59:11.987864 10089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395095474643-392-0/minicluster-data/master-0/wal/instance:
uuid: "464529f3459d46438c1d94067da12d1c"
format_stamp: "Formatted at 2026-09-26 03:59:11 on dist-test-slave-nhpw"
server_key: "97293130fc04ec9d8cbbac92f0cd0087"
server_key_iv: "03e23e446881744c79896a4e1a67ad8f"
server_key_version: "encryptionkey@0"
I20260926 03:59:11.989318 10089 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 03:59:11.990041 10103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:59:11.990247 10089 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 03:59:11.990363 10089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395095474643-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395095474643-392-0/minicluster-data/master-0/wal
uuid: "464529f3459d46438c1d94067da12d1c"
format_stamp: "Formatted at 2026-09-26 03:59:11 on dist-test-slave-nhpw"
server_key: "97293130fc04ec9d8cbbac92f0cd0087"
server_key_iv: "03e23e446881744c79896a4e1a67ad8f"
server_key_version: "encryptionkey@0"
I20260926 03:59:11.990444 10089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395095474643-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395095474643-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395095474643-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:59:11.993878 10089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:59:11.994153 10089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:59:11.994294 10089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:59:11.998155 10089 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42213
I20260926 03:59:11.998188 10155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42213 every 8 connection(s)
I20260926 03:59:11.998522 10089 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
I20260926 03:59:11.999197 10156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:59:12.002065 10156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 464529f3459d46438c1d94067da12d1c: Bootstrap starting.
I20260926 03:59:12.002674 10156 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 464529f3459d46438c1d94067da12d1c: Neither blocks nor log segments found. Creating new log.
I20260926 03:59:12.002988 10156 log.cc:824] T 00000000000000000000000000000000 P 464529f3459d46438c1d94067da12d1c: Log is configured to *not* fsync() on all Append() calls
I20260926 03:59:12.003716 10156 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 464529f3459d46438c1d94067da12d1c: No bootstrap required, opened a new log
I20260926 03:59:12.005430 10156 raft_consensus.cc:348] T 00000000000000000000000000000000 P 464529f3459d46438c1d94067da12d1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "464529f3459d46438c1d94067da12d1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42213 } }
I20260926 03:59:12.005602 10156 raft_consensus.cc:374] T 00000000000000000000000000000000 P 464529f3459d46438c1d94067da12d1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:59:12.005653 10156 raft_consensus.cc:729] T 00000000000000000000000000000000 P 464529f3459d46438c1d94067da12d1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 464529f3459d46438c1d94067da12d1c, State: Initialized, Role: FOLLOWER
I20260926 03:59:12.005728 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 10089
I20260926 03:59:12.005825 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395095474643-392-0/minicluster-data/master-0/wal/instance
I20260926 03:59:12.005884 10156 consensus_queue.cc:260] T 00000000000000000000000000000000 P 464529f3459d46438c1d94067da12d1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "464529f3459d46438c1d94067da12d1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42213 } }
I20260926 03:59:12.005987 10156 raft_consensus.cc:388] T 00000000000000000000000000000000 P 464529f3459d46438c1d94067da12d1c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:59:12.006019 392 external_mini_cluster.cc:1428] Setting key bd031b1ad62ec6b7a69186b8dae72aad
I20260926 03:59:12.006032 10156 raft_consensus.cc:482] T 00000000000000000000000000000000 P 464529f3459d46438c1d94067da12d1c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:59:12.006078 10156 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 464529f3459d46438c1d94067da12d1c [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:59:12.006861 10156 raft_consensus.cc:504] T 00000000000000000000000000000000 P 464529f3459d46438c1d94067da12d1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "464529f3459d46438c1d94067da12d1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42213 } }
I20260926 03:59:12.006971 10156 leader_election.cc:302] T 00000000000000000000000000000000 P 464529f3459d46438c1d94067da12d1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 464529f3459d46438c1d94067da12d1c; no voters:
I20260926 03:59:12.007125 10156 leader_election.cc:288] T 00000000000000000000000000000000 P 464529f3459d46438c1d94067da12d1c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:59:12.007216 10161 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 464529f3459d46438c1d94067da12d1c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:59:12.007397 10161 raft_consensus.cc:686] T 00000000000000000000000000000000 P 464529f3459d46438c1d94067da12d1c [term 1 LEADER]: Becoming Leader. State: Replica: 464529f3459d46438c1d94067da12d1c, State: Running, Role: LEADER
I20260926 03:59:12.007400 10156 sys_catalog.cc:564] T 00000000000000000000000000000000 P 464529f3459d46438c1d94067da12d1c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:59:12.007512 10161 consensus_queue.cc:237] T 00000000000000000000000000000000 P 464529f3459d46438c1d94067da12d1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "464529f3459d46438c1d94067da12d1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42213 } }
I20260926 03:59:12.008222 10162 sys_catalog.cc:455] T 00000000000000000000000000000000 P 464529f3459d46438c1d94067da12d1c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "464529f3459d46438c1d94067da12d1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "464529f3459d46438c1d94067da12d1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42213 } } }
I20260926 03:59:12.008308 10162 sys_catalog.cc:458] T 00000000000000000000000000000000 P 464529f3459d46438c1d94067da12d1c [sys.catalog]: This master's current role is: LEADER
I20260926 03:59:12.008453 10162 sys_catalog.cc:455] T 00000000000000000000000000000000 P 464529f3459d46438c1d94067da12d1c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 464529f3459d46438c1d94067da12d1c. Latest consensus state: current_term: 1 leader_uuid: "464529f3459d46438c1d94067da12d1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "464529f3459d46438c1d94067da12d1c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42213 } } }
I20260926 03:59:12.008481 10160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:12.006527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53552 (local address 127.0.98.62:42213)
0926 03:59:12.006775 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0926 03:59:12.006785 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:59:12.006821 (+ 36us) server_negotiation.cc:407] Connection header received
0926 03:59:12.006943 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:59:12.006951 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:59:12.006999 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:59:12.007158 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:59:12.007953 (+ 795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:59:12.007958 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:59:12.007958 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:59:12.007964 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:59:12.008015 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:59:12.008030 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:59:12.008070 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0926 03:59:12.008215 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 03:59:12.008279 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":105,"threads_started":1}
I20260926 03:59:12.008653 10162 sys_catalog.cc:458] T 00000000000000000000000000000000 P 464529f3459d46438c1d94067da12d1c [sys.catalog]: This master's current role is: LEADER
W20260926 03:59:12.009330 10176 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 464529f3459d46438c1d94067da12d1c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 03:59:12.009379 10176 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 03:59:12.009572 10170 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:59:12.009793 10170 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:59:12.011193 10170 catalog_manager.cc:1347] Generated new cluster ID: 635055e73579436ebcb0c8e073576819
I20260926 03:59:12.011243 10170 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:59:12.038194 10170 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:59:12.038429 10170 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:59:12.048414 10170 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 464529f3459d46438c1d94067da12d1c: Generated new TSK 0
I20260926 03:59:12.048668 10170 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 03:59:12.055560 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395095474643-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395095474643-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395095474643-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42213
--builtin_ntp_servers=127.0.98.20:35513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 03:59:12.143361 10180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:59:12.143575 10180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:59:12.143599 10180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:59:12.144919 10180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:59:12.144963 10180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:59:12.145010 10180 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 03:59:12.146377 10180 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395095474643-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395095474643-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10180
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395095474643-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:59:12.146631 10180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:59:12.146930 10180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:59:12.151886 10185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:59:12.151895 10186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:59:12.151995 10180 server_base.cc:1034] running on GCE node
W20260926 03:59:12.151938 10188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:59:12.152340 10180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:59:12.152539 10180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:59:12.153677 10180 hybrid_clock.cc:648] HybridClock initialized: now 1790395152153678 us; error 44 us; skew 500 ppm
I20260926 03:59:12.154951 10180 webserver.cc:460] Webserver started at http://127.0.98.1:43763/ using document root <none> and password file <none>
I20260926 03:59:12.155179 10180 fs_manager.cc:359] Metadata directory not provided
I20260926 03:59:12.155224 10180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:59:12.155390 10180 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:59:12.156456 10180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395095474643-392-0/minicluster-data/ts-0/data/instance:
uuid: "baf51626eccb40bc8311fac9d0c65b1d"
format_stamp: "Formatted at 2026-09-26 03:59:12 on dist-test-slave-nhpw"
server_key: "86a134ab2ef5313bcac145975b97c1ee"
server_key_iv: "42f6a5a7a28449ed80b7a9e301b1c2a2"
server_key_version: "encryptionkey@0"
I20260926 03:59:12.156844 10180 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395095474643-392-0/minicluster-data/ts-0/wal/instance:
uuid: "baf51626eccb40bc8311fac9d0c65b1d"
format_stamp: "Formatted at 2026-09-26 03:59:12 on dist-test-slave-nhpw"
server_key: "86a134ab2ef5313bcac145975b97c1ee"
server_key_iv: "42f6a5a7a28449ed80b7a9e301b1c2a2"
server_key_version: "encryptionkey@0"
I20260926 03:59:12.158139 10180 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 03:59:12.158840 10194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:59:12.159035 10180 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 03:59:12.159132 10180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395095474643-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395095474643-392-0/minicluster-data/ts-0/wal
uuid: "baf51626eccb40bc8311fac9d0c65b1d"
format_stamp: "Formatted at 2026-09-26 03:59:12 on dist-test-slave-nhpw"
server_key: "86a134ab2ef5313bcac145975b97c1ee"
server_key_iv: "42f6a5a7a28449ed80b7a9e301b1c2a2"
server_key_version: "encryptionkey@0"
I20260926 03:59:12.159211 10180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395095474643-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395095474643-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395095474643-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:59:12.162515 10180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:59:12.162767 10180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:59:12.162915 10180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:59:12.163120 10180 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:59:12.163414 10180 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:59:12.163467 10180 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:59:12.163507 10180 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:59:12.163534 10180 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:59:12.169273 10180 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40245
I20260926 03:59:12.169342 10307 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40245 every 8 connection(s)
I20260926 03:59:12.169668 10180 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
I20260926 03:59:12.170796 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 10180
I20260926 03:59:12.170874 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395095474643-392-0/minicluster-data/ts-0/wal/instance
I20260926 03:59:12.171083 392 external_mini_cluster.cc:1428] Setting key ac8b1e8104df1b11e0eb6fbd71bdebc4
I20260926 03:59:12.172142 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395095474643-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395095474643-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395095474643-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42213
--builtin_ntp_servers=127.0.98.20:35513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 03:59:12.173256 10160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:12.170423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33501 (local address 127.0.98.62:42213)
0926 03:59:12.170491 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 03:59:12.170493 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:59:12.170727 (+ 234us) server_negotiation.cc:407] Connection header received
0926 03:59:12.171429 (+ 702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:59:12.171430 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:59:12.171438 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:59:12.171511 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:59:12.172986 (+ 1475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:59:12.172987 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:59:12.172988 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:59:12.172992 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:59:12.173010 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:59:12.173015 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:59:12.173071 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 03:59:12.173169 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 03:59:12.173187 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 03:59:12.173467 10310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:12.170400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42213 (local address 127.0.98.1:33501)
0926 03:59:12.170598 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 03:59:12.170631 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 03:59:12.171280 (+ 649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:59:12.171592 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:59:12.171596 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:59:12.171606 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:59:12.172809 (+ 1203us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:59:12.172809 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:59:12.172838 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:59:12.172874 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:59:12.173074 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:59:12.173079 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:59:12.173082 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 03:59:12.173179 (+ 97us) client_negotiation.cc:240] Negotiation successful
0926 03:59:12.173244 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":85,"threads_started":1}
I20260926 03:59:12.173972 10308 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42213
I20260926 03:59:12.174078 10308 heartbeater.cc:464] Registering TS with master...
I20260926 03:59:12.174234 10308 heartbeater.cc:511] Master 127.0.98.62:42213 requested a full tablet report, sending...
I20260926 03:59:12.174841 10120 ts_manager.cc:194] Registered new tserver with Master: baf51626eccb40bc8311fac9d0c65b1d (127.0.98.1:40245)
W20260926 03:59:12.253412 10311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:59:12.253628 10311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:59:12.253650 10311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:59:12.254745 10311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:59:12.254781 10311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:59:12.254805 10311 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 03:59:12.256074 10311 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395095474643-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395095474643-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10311
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395095474643-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:59:12.256278 10311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:59:12.256517 10311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:59:12.261911 10319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:59:12.261958 10316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:59:12.261965 10317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:59:12.262013 10311 server_base.cc:1034] running on GCE node
I20260926 03:59:12.262300 10311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:59:12.262567 10311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:59:12.263734 10311 hybrid_clock.cc:648] HybridClock initialized: now 1790395152263698 us; error 49 us; skew 500 ppm
I20260926 03:59:12.265009 10311 webserver.cc:460] Webserver started at http://127.0.98.2:33239/ using document root <none> and password file <none>
I20260926 03:59:12.265249 10311 fs_manager.cc:359] Metadata directory not provided
I20260926 03:59:12.265319 10311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:59:12.265522 10311 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:59:12.266607 10311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395095474643-392-0/minicluster-data/ts-1/data/instance:
uuid: "9308772e2e984f4bb9e2b1826193967e"
format_stamp: "Formatted at 2026-09-26 03:59:12 on dist-test-slave-nhpw"
server_key: "a3fcbc87347d356f69b89babd943a5d4"
server_key_iv: "136707603efab85386cb3ea25a15a15a"
server_key_version: "encryptionkey@0"
I20260926 03:59:12.266979 10311 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395095474643-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9308772e2e984f4bb9e2b1826193967e"
format_stamp: "Formatted at 2026-09-26 03:59:12 on dist-test-slave-nhpw"
server_key: "a3fcbc87347d356f69b89babd943a5d4"
server_key_iv: "136707603efab85386cb3ea25a15a15a"
server_key_version: "encryptionkey@0"
I20260926 03:59:12.268294 10311 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 03:59:12.269126 10325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:59:12.269301 10311 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 03:59:12.269419 10311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395095474643-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395095474643-392-0/minicluster-data/ts-1/wal
uuid: "9308772e2e984f4bb9e2b1826193967e"
format_stamp: "Formatted at 2026-09-26 03:59:12 on dist-test-slave-nhpw"
server_key: "a3fcbc87347d356f69b89babd943a5d4"
server_key_iv: "136707603efab85386cb3ea25a15a15a"
server_key_version: "encryptionkey@0"
I20260926 03:59:12.269497 10311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395095474643-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395095474643-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395095474643-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:59:12.272744 10311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:59:12.273022 10311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:59:12.273164 10311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:59:12.273371 10311 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:59:12.273674 10311 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:59:12.273727 10311 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:59:12.273765 10311 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:59:12.273794 10311 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:59:12.279853 10311 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36125
I20260926 03:59:12.279906 10438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36125 every 8 connection(s)
I20260926 03:59:12.280207 10311 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
I20260926 03:59:12.283278 10160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:12.280926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58815 (local address 127.0.98.62:42213)
0926 03:59:12.280995 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 03:59:12.280997 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:59:12.281240 (+ 243us) server_negotiation.cc:407] Connection header received
0926 03:59:12.281757 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:59:12.281758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:59:12.281765 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:59:12.281823 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:59:12.282992 (+ 1169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:59:12.282994 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:59:12.282995 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:59:12.283000 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:59:12.283016 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:59:12.283021 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:59:12.283099 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 03:59:12.283178 (+ 79us) server_negotiation.cc:299] Negotiation successful
0926 03:59:12.283194 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 03:59:12.283491 10441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:12.280898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42213 (local address 127.0.98.2:58815)
0926 03:59:12.281115 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0926 03:59:12.281158 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 03:59:12.281647 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:59:12.281900 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:59:12.281905 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:59:12.281914 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:59:12.282821 (+ 907us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:59:12.282821 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:59:12.282849 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:59:12.282877 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:59:12.283097 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:59:12.283103 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:59:12.283106 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 03:59:12.283215 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 03:59:12.283279 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":123,"threads_started":1}
I20260926 03:59:12.283982 10439 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42213
I20260926 03:59:12.284091 10439 heartbeater.cc:464] Registering TS with master...
I20260926 03:59:12.284243 10439 heartbeater.cc:511] Master 127.0.98.62:42213 requested a full tablet report, sending...
I20260926 03:59:12.284575 10120 ts_manager.cc:194] Registered new tserver with Master: 9308772e2e984f4bb9e2b1826193967e (127.0.98.2:36125)
I20260926 03:59:12.287329 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 10311
I20260926 03:59:12.287405 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395095474643-392-0/minicluster-data/ts-1/wal/instance
I20260926 03:59:12.287541 392 external_mini_cluster.cc:1428] Setting key 89d696ad1e571f454392b181f3698ffe
I20260926 03:59:12.288376 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395095474643-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395095474643-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395095474643-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42213
--builtin_ntp_servers=127.0.98.20:35513
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 03:59:12.368105 10442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:59:12.368311 10442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:59:12.368332 10442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:59:12.369452 10442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:59:12.369488 10442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:59:12.369519 10442 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 03:59:12.370766 10442 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35513
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395095474643-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395095474643-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10442
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395095474643-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:59:12.370950 10442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:59:12.371196 10442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:59:12.376371 10450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:59:12.376426 10448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:59:12.376401 10442 server_base.cc:1034] running on GCE node
W20260926 03:59:12.376371 10447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:59:12.376803 10442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:59:12.377054 10442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:59:12.378232 10442 hybrid_clock.cc:648] HybridClock initialized: now 1790395152378222 us; error 33 us; skew 500 ppm
I20260926 03:59:12.379510 10442 webserver.cc:460] Webserver started at http://127.0.98.3:45625/ using document root <none> and password file <none>
I20260926 03:59:12.379769 10442 fs_manager.cc:359] Metadata directory not provided
I20260926 03:59:12.379837 10442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:59:12.379994 10442 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:59:12.381067 10442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395095474643-392-0/minicluster-data/ts-2/data/instance:
uuid: "de3810e5a9904b9abab648982c2afaf3"
format_stamp: "Formatted at 2026-09-26 03:59:12 on dist-test-slave-nhpw"
server_key: "00d8585d017f0c5d08c039202bc59bb4"
server_key_iv: "0288aab27fc0abb0e1ad6675cb410c4d"
server_key_version: "encryptionkey@0"
I20260926 03:59:12.381422 10442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395095474643-392-0/minicluster-data/ts-2/wal/instance:
uuid: "de3810e5a9904b9abab648982c2afaf3"
format_stamp: "Formatted at 2026-09-26 03:59:12 on dist-test-slave-nhpw"
server_key: "00d8585d017f0c5d08c039202bc59bb4"
server_key_iv: "0288aab27fc0abb0e1ad6675cb410c4d"
server_key_version: "encryptionkey@0"
I20260926 03:59:12.382686 10442 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 03:59:12.383405 10456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:59:12.383564 10442 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 03:59:12.383656 10442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395095474643-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395095474643-392-0/minicluster-data/ts-2/wal
uuid: "de3810e5a9904b9abab648982c2afaf3"
format_stamp: "Formatted at 2026-09-26 03:59:12 on dist-test-slave-nhpw"
server_key: "00d8585d017f0c5d08c039202bc59bb4"
server_key_iv: "0288aab27fc0abb0e1ad6675cb410c4d"
server_key_version: "encryptionkey@0"
I20260926 03:59:12.383728 10442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395095474643-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395095474643-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395095474643-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:59:12.386935 10442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:59:12.387177 10442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:59:12.387310 10442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:59:12.387497 10442 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:59:12.387784 10442 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:59:12.387815 10442 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:59:12.387867 10442 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:59:12.387923 10442 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:59:12.393572 10442 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42303
I20260926 03:59:12.393651 10569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42303 every 8 connection(s)
I20260926 03:59:12.393970 10442 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
I20260926 03:59:12.396729 10160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:12.394661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51677 (local address 127.0.98.62:42213)
0926 03:59:12.394724 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 03:59:12.394726 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:59:12.394954 (+ 228us) server_negotiation.cc:407] Connection header received
0926 03:59:12.395353 (+ 399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:59:12.395371 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:59:12.395376 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:59:12.395429 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:59:12.396476 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:59:12.396477 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:59:12.396478 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 03:59:12.396482 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:59:12.396495 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 03:59:12.396499 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:59:12.396553 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 03:59:12.396614 (+ 61us) server_negotiation.cc:299] Negotiation successful
0926 03:59:12.396627 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 03:59:12.396931 10572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:12.394664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42213 (local address 127.0.98.3:51677)
0926 03:59:12.394824 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 03:59:12.394855 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 03:59:12.395274 (+ 419us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:59:12.395478 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:59:12.395481 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:59:12.395488 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:59:12.396336 (+ 848us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 03:59:12.396337 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:59:12.396362 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 03:59:12.396388 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:59:12.396553 (+ 165us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:59:12.396555 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:59:12.396557 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 03:59:12.396608 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 03:59:12.396701 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":70,"threads_started":1}
I20260926 03:59:12.397362 10570 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42213
I20260926 03:59:12.397454 10570 heartbeater.cc:464] Registering TS with master...
I20260926 03:59:12.397619 10570 heartbeater.cc:511] Master 127.0.98.62:42213 requested a full tablet report, sending...
I20260926 03:59:12.397997 10120 ts_manager.cc:194] Registered new tserver with Master: de3810e5a9904b9abab648982c2afaf3 (127.0.98.3:42303)
I20260926 03:59:12.402632 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 10442
I20260926 03:59:12.402738 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790395095474643-392-0/minicluster-data/ts-2/wal/instance
I20260926 03:59:12.402930 392 external_mini_cluster.cc:1428] Setting key 2af272772b55267722ea130a01efb19e
I20260926 03:59:12.403761 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 03:59:12.405519 10160 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 03:59:12.405083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42860 (local address 127.0.98.62:42213)
0926 03:59:12.405145 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 03:59:12.405147 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:59:12.405233 (+ 86us) server_negotiation.cc:407] Connection header received
0926 03:59:12.405290 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:59:12.405290 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:59:12.405294 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:59:12.405348 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:59:12.405439 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42860: BlockingRecv error: recv got EOF from 127.0.0.1:42860 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 03:59:12.405618 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 10180
I20260926 03:59:12.410379 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 10311
I20260926 03:59:12.414696 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 10442
I20260926 03:59:12.418668 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 10089
2026-09-26T03:59:12Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (553 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:59:12 dist-test-slave-nhpw krb5kdc[10580](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:59:12 dist-test-slave-nhpw krb5kdc[10580](info): set up 2 sockets
Sep 26 03:59:12 dist-test-slave-nhpw krb5kdc[10580](info): commencing operation
krb5kdc: starting...
W20260926 03:59:13.453570 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/krb5kdc/test-user.keytab.
Sep 26 03:59:13 dist-test-slave-nhpw krb5kdc[10580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395153, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:59:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:59:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 03:59:13.577399 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43501
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43501
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:59:13.659538 10593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:59:13.659763 10593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:59:13.659786 10593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:59:13.661055 10593 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:59:13.661108 10593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:59:13.661123 10593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:59:13.661136 10593 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:59:13.661149 10593 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:59:13.662701 10593 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43501
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:59:13.662961 10593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:59:13.663221 10593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:59:13.668212 10601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:59:13.668332 10598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:59:13.668567 10593 server_base.cc:1034] running on GCE node
W20260926 03:59:13.668691 10599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:59:13.668937 10593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:59:13.669188 10593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:59:13.670370 10593 hybrid_clock.cc:648] HybridClock initialized: now 1790395153670363 us; error 38 us; skew 500 ppm
Sep 26 03:59:13 dist-test-slave-nhpw krb5kdc[10580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395153, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:59:13.673156 10593 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 03:59:13.673619 10593 webserver.cc:460] Webserver started at http://127.0.98.62:45421/ using document root <none> and password file <none>
I20260926 03:59:13.673877 10593 fs_manager.cc:359] Metadata directory not provided
I20260926 03:59:13.673945 10593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:59:13.674093 10593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:59:13.675181 10593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/minicluster-data/master-0/data/instance:
uuid: "1423b97378254cd5ae147f3fb9db4c51"
format_stamp: "Formatted at 2026-09-26 03:59:13 on dist-test-slave-nhpw"
server_key: "a6b989df375c039410451ab816b018ca"
server_key_iv: "09df1034fb80f0a6f11f31827da0744a"
server_key_version: "encryptionkey@0"
I20260926 03:59:13.675551 10593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/minicluster-data/master-0/wal/instance:
uuid: "1423b97378254cd5ae147f3fb9db4c51"
format_stamp: "Formatted at 2026-09-26 03:59:13 on dist-test-slave-nhpw"
server_key: "a6b989df375c039410451ab816b018ca"
server_key_iv: "09df1034fb80f0a6f11f31827da0744a"
server_key_version: "encryptionkey@0"
I20260926 03:59:13.676981 10593 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 03:59:13.677692 10608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:59:13.677878 10593 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 03:59:13.677984 10593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/minicluster-data/master-0/wal
uuid: "1423b97378254cd5ae147f3fb9db4c51"
format_stamp: "Formatted at 2026-09-26 03:59:13 on dist-test-slave-nhpw"
server_key: "a6b989df375c039410451ab816b018ca"
server_key_iv: "09df1034fb80f0a6f11f31827da0744a"
server_key_version: "encryptionkey@0"
I20260926 03:59:13.678058 10593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:59:13.695899 10593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:59:13.698318 10593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:59:13.698482 10593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:59:13.702273 10593 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43501
I20260926 03:59:13.702311 10660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43501 every 8 connection(s)
I20260926 03:59:13.702670 10593 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
I20260926 03:59:13.703673 10661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:59:13.706018 10661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1423b97378254cd5ae147f3fb9db4c51: Bootstrap starting.
I20260926 03:59:13.706586 10661 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1423b97378254cd5ae147f3fb9db4c51: Neither blocks nor log segments found. Creating new log.
I20260926 03:59:13.706879 10661 log.cc:824] T 00000000000000000000000000000000 P 1423b97378254cd5ae147f3fb9db4c51: Log is configured to *not* fsync() on all Append() calls
I20260926 03:59:13.707562 10661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1423b97378254cd5ae147f3fb9db4c51: No bootstrap required, opened a new log
I20260926 03:59:13.709128 10661 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1423b97378254cd5ae147f3fb9db4c51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1423b97378254cd5ae147f3fb9db4c51" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43501 } }
I20260926 03:59:13.709260 10661 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1423b97378254cd5ae147f3fb9db4c51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:59:13.709282 10661 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1423b97378254cd5ae147f3fb9db4c51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1423b97378254cd5ae147f3fb9db4c51, State: Initialized, Role: FOLLOWER
I20260926 03:59:13.709465 10661 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1423b97378254cd5ae147f3fb9db4c51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1423b97378254cd5ae147f3fb9db4c51" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43501 } }
I20260926 03:59:13.709529 10661 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1423b97378254cd5ae147f3fb9db4c51 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:59:13.709550 10661 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1423b97378254cd5ae147f3fb9db4c51 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:59:13.709616 10661 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1423b97378254cd5ae147f3fb9db4c51 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:59:13.710379 10661 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1423b97378254cd5ae147f3fb9db4c51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1423b97378254cd5ae147f3fb9db4c51" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43501 } }
I20260926 03:59:13.710491 10661 leader_election.cc:302] T 00000000000000000000000000000000 P 1423b97378254cd5ae147f3fb9db4c51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1423b97378254cd5ae147f3fb9db4c51; no voters:
I20260926 03:59:13.710661 10661 leader_election.cc:288] T 00000000000000000000000000000000 P 1423b97378254cd5ae147f3fb9db4c51 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:59:13.710703 10664 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1423b97378254cd5ae147f3fb9db4c51 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:59:13.710856 10664 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1423b97378254cd5ae147f3fb9db4c51 [term 1 LEADER]: Becoming Leader. State: Replica: 1423b97378254cd5ae147f3fb9db4c51, State: Running, Role: LEADER
I20260926 03:59:13.710927 10661 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1423b97378254cd5ae147f3fb9db4c51 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:59:13.710939 10664 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1423b97378254cd5ae147f3fb9db4c51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1423b97378254cd5ae147f3fb9db4c51" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43501 } }
I20260926 03:59:13.711323 10665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1423b97378254cd5ae147f3fb9db4c51 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1423b97378254cd5ae147f3fb9db4c51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1423b97378254cd5ae147f3fb9db4c51" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43501 } } }
I20260926 03:59:13.711330 10666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1423b97378254cd5ae147f3fb9db4c51 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1423b97378254cd5ae147f3fb9db4c51. Latest consensus state: current_term: 1 leader_uuid: "1423b97378254cd5ae147f3fb9db4c51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1423b97378254cd5ae147f3fb9db4c51" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43501 } } }
I20260926 03:59:13.711423 10665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1423b97378254cd5ae147f3fb9db4c51 [sys.catalog]: This master's current role is: LEADER
I20260926 03:59:13.711427 10666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1423b97378254cd5ae147f3fb9db4c51 [sys.catalog]: This master's current role is: LEADER
I20260926 03:59:13.711645 10672 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:59:13.712229 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 10593
I20260926 03:59:13.712329 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/minicluster-data/master-0/wal/instance
I20260926 03:59:13.712549 392 external_mini_cluster.cc:1428] Setting key 8c93a3f51d7629be3a6f30923c9a32e0
I20260926 03:59:13.712882 10672 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 03:59:13.714453 10672 catalog_manager.cc:1347] Generated new cluster ID: ae48fffbd19f4431a4ac3d8cb69a01c0
I20260926 03:59:13.714511 10672 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 03:59:13 dist-test-slave-nhpw krb5kdc[10580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395153, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 03:59:13.729326 10672 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:59:13.729955 10672 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:59:13.730156 10682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:13.713020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43624 (local address 127.0.98.62:43501)
0926 03:59:13.713190 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 03:59:13.713200 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 03:59:13.720886 (+ 7686us) server_negotiation.cc:407] Connection header received
0926 03:59:13.721050 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:59:13.721055 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:59:13.721101 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:59:13.721233 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:59:13.721871 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:13.722791 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:13.723565 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:13.723963 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:13.724094 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:13.726480 (+ 2386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:59:13.726484 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:59:13.726484 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:59:13.726496 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:59:13.728190 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:59:13.728575 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:59:13.728578 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:59:13.728578 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:59:13.728735 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:59:13.728934 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:59:13.728935 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:59:13.728936 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:59:13.729220 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:59:13.729393 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0926 03:59:13.729643 (+ 250us) server_negotiation.cc:299] Negotiation successful
0926 03:59:13.729928 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":93,"threads_started":1}
I20260926 03:59:13.732765 10672 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1423b97378254cd5ae147f3fb9db4c51: Generated new TSK 0
I20260926 03:59:13.732930 10672 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 03:59:13.760146 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43501
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:59:13.840878 10687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:59:13.841133 10687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:59:13.841172 10687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:59:13.842363 10687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:59:13.842439 10687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:59:13.842486 10687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 03:59:13.843988 10687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:59:13.844264 10687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:59:13.844553 10687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:59:13.849617 10695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:59:13.849670 10693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:59:13.849620 10692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:59:13.849880 10687 server_base.cc:1034] running on GCE node
I20260926 03:59:13.850109 10687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:59:13.850412 10687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:59:13.851603 10687 hybrid_clock.cc:648] HybridClock initialized: now 1790395153851593 us; error 38 us; skew 500 ppm
Sep 26 03:59:13 dist-test-slave-nhpw krb5kdc[10580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395153, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:59:13.854259 10687 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 03:59:13.854766 10687 webserver.cc:460] Webserver started at http://127.0.98.1:41165/ using document root <none> and password file <none>
I20260926 03:59:13.855005 10687 fs_manager.cc:359] Metadata directory not provided
I20260926 03:59:13.855077 10687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:59:13.855229 10687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:59:13.856297 10687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/minicluster-data/ts-0/data/instance:
uuid: "1958d3dc405249bc9721c7dab0cff947"
format_stamp: "Formatted at 2026-09-26 03:59:13 on dist-test-slave-nhpw"
server_key: "cc280545dc2006354b959e8e4db080b1"
server_key_iv: "5967fac464aa3eed5719a1670ce94bcb"
server_key_version: "encryptionkey@0"
I20260926 03:59:13.856741 10687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1958d3dc405249bc9721c7dab0cff947"
format_stamp: "Formatted at 2026-09-26 03:59:13 on dist-test-slave-nhpw"
server_key: "cc280545dc2006354b959e8e4db080b1"
server_key_iv: "5967fac464aa3eed5719a1670ce94bcb"
server_key_version: "encryptionkey@0"
I20260926 03:59:13.858071 10687 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 03:59:13.858851 10702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:59:13.859019 10687 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 03:59:13.859117 10687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/minicluster-data/ts-0/wal
uuid: "1958d3dc405249bc9721c7dab0cff947"
format_stamp: "Formatted at 2026-09-26 03:59:13 on dist-test-slave-nhpw"
server_key: "cc280545dc2006354b959e8e4db080b1"
server_key_iv: "5967fac464aa3eed5719a1670ce94bcb"
server_key_version: "encryptionkey@0"
I20260926 03:59:13.859202 10687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:59:13.871650 10687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:59:13.873653 10687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:59:13.873816 10687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:59:13.874068 10687 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:59:13.874325 10687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:59:13.874356 10687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:59:13.874384 10687 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:59:13.874397 10687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:59:13.880067 10687 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41103
I20260926 03:59:13.880154 10815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41103 every 8 connection(s)
I20260926 03:59:13.880476 10687 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
I20260926 03:59:13.885079 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 10687
I20260926 03:59:13.885190 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/minicluster-data/ts-0/wal/instance
I20260926 03:59:13.885373 392 external_mini_cluster.cc:1428] Setting key e6022f6ff60a2c1f61bfb4a4679aaa9b
Sep 26 03:59:13 dist-test-slave-nhpw krb5kdc[10580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395153, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 03:59:13.889853 10682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:13.881262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51219 (local address 127.0.98.62:43501)
0926 03:59:13.881319 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 03:59:13.881321 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:59:13.881543 (+ 222us) server_negotiation.cc:407] Connection header received
0926 03:59:13.882075 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:59:13.882076 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:59:13.882089 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:59:13.882143 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:59:13.883028 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:13.883532 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:13.884470 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:13.884725 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:13.884785 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:13.886885 (+ 2100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:59:13.886887 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:59:13.886887 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:59:13.886895 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:59:13.888469 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:59:13.888890 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:59:13.888892 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:59:13.888892 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:59:13.888955 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:59:13.889180 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:59:13.889181 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:59:13.889181 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:59:13.889330 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:59:13.889414 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 03:59:13.889630 (+ 216us) server_negotiation.cc:299] Negotiation successful
0926 03:59:13.889727 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 03:59:13.889869 10818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:13.881267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43501 (local address 127.0.98.1:51219)
0926 03:59:13.881422 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 03:59:13.881456 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 03:59:13.881990 (+ 534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:59:13.882195 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:59:13.882199 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:59:13.882760 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:59:13.882953 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:13.882958 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:13.883612 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:13.883613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:13.884396 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:13.884398 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:13.884792 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:13.884793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:13.884893 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:13.885435 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:59:13.885458 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:59:13.886796 (+ 1338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:59:13.888543 (+ 1747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:59:13.888548 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:59:13.888550 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:59:13.888821 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:59:13.889023 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:59:13.889024 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:59:13.889024 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:59:13.889117 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:59:13.889401 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:59:13.889406 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:59:13.889518 (+ 112us) client_negotiation.cc:769] Sending connection context
0926 03:59:13.889597 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 03:59:13.889716 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":94,"threads_started":1}
I20260926 03:59:13.890513 10816 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43501
I20260926 03:59:13.890645 10816 heartbeater.cc:464] Registering TS with master...
I20260926 03:59:13.890839 10816 heartbeater.cc:511] Master 127.0.98.62:43501 requested a full tablet report, sending...
I20260926 03:59:13.891484 10625 ts_manager.cc:194] Registered new tserver with Master: 1958d3dc405249bc9721c7dab0cff947 (127.0.98.1:41103)
I20260926 03:59:13.892223 10625 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:51219
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 03:59:13.912205 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43501
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:59:13.993247 10821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:59:13.993458 10821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:59:13.993481 10821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:59:13.994544 10821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:59:13.994577 10821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:59:13.994602 10821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 03:59:13.996021 10821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:59:13.996219 10821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:59:13.996455 10821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:59:14.001698 10829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:59:14.001854 10821 server_base.cc:1034] running on GCE node
W20260926 03:59:14.001699 10826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:59:14.001718 10827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:59:14.002161 10821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:59:14.002360 10821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:59:14.003520 10821 hybrid_clock.cc:648] HybridClock initialized: now 1790395154003504 us; error 45 us; skew 500 ppm
Sep 26 03:59:14 dist-test-slave-nhpw krb5kdc[10580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395154, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:59:14.006039 10821 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 03:59:14.006537 10821 webserver.cc:460] Webserver started at http://127.0.98.2:42333/ using document root <none> and password file <none>
I20260926 03:59:14.006752 10821 fs_manager.cc:359] Metadata directory not provided
I20260926 03:59:14.006816 10821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:59:14.006958 10821 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:59:14.007988 10821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/minicluster-data/ts-1/data/instance:
uuid: "983d20573bf54a119bda135144b9a4af"
format_stamp: "Formatted at 2026-09-26 03:59:14 on dist-test-slave-nhpw"
server_key: "66945d376eca5b33382cb16e386a6d99"
server_key_iv: "32a97fa3b0de6aa2e381eae5ccffe05c"
server_key_version: "encryptionkey@0"
I20260926 03:59:14.008354 10821 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/minicluster-data/ts-1/wal/instance:
uuid: "983d20573bf54a119bda135144b9a4af"
format_stamp: "Formatted at 2026-09-26 03:59:14 on dist-test-slave-nhpw"
server_key: "66945d376eca5b33382cb16e386a6d99"
server_key_iv: "32a97fa3b0de6aa2e381eae5ccffe05c"
server_key_version: "encryptionkey@0"
I20260926 03:59:14.010092 10821 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260926 03:59:14.010887 10836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:59:14.011049 10821 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 03:59:14.011137 10821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/minicluster-data/ts-1/wal
uuid: "983d20573bf54a119bda135144b9a4af"
format_stamp: "Formatted at 2026-09-26 03:59:14 on dist-test-slave-nhpw"
server_key: "66945d376eca5b33382cb16e386a6d99"
server_key_iv: "32a97fa3b0de6aa2e381eae5ccffe05c"
server_key_version: "encryptionkey@0"
I20260926 03:59:14.011193 10821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:59:14.023058 10821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:59:14.025455 10821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:59:14.025571 10821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:59:14.025754 10821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:59:14.026075 10821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:59:14.026113 10821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:59:14.026136 10821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:59:14.026185 10821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:59:14.031807 10821 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34011
I20260926 03:59:14.031888 10949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34011 every 8 connection(s)
I20260926 03:59:14.032204 10821 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
I20260926 03:59:14.036996 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 10821
I20260926 03:59:14.037113 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/minicluster-data/ts-1/wal/instance
I20260926 03:59:14.037319 392 external_mini_cluster.cc:1428] Setting key 4cbe771d44e0711912069b44124047b3
Sep 26 03:59:14 dist-test-slave-nhpw krb5kdc[10580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395154, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 03:59:14.042640 10682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:14.033064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39539 (local address 127.0.98.62:43501)
0926 03:59:14.033150 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0926 03:59:14.033152 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:59:14.033361 (+ 209us) server_negotiation.cc:407] Connection header received
0926 03:59:14.033835 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:59:14.033837 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:59:14.033847 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:59:14.033914 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:59:14.034878 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:14.035416 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:14.036427 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:14.036623 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:14.036767 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:14.039519 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:59:14.039520 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:59:14.039521 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:59:14.039527 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:59:14.040930 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:59:14.041533 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:59:14.041534 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:59:14.041535 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:59:14.041595 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:59:14.041941 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:59:14.041943 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:59:14.041944 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:59:14.042094 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:59:14.042156 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 03:59:14.042504 (+ 348us) server_negotiation.cc:299] Negotiation successful
0926 03:59:14.042575 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 03:59:14.042969 10952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:14.033035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43501 (local address 127.0.98.2:39539)
0926 03:59:14.033205 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 03:59:14.033249 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 03:59:14.033734 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:59:14.033995 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:59:14.033999 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:59:14.034570 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:59:14.034765 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:14.034769 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:14.035508 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:14.035509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:14.036328 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:14.036330 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:14.036777 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:14.036778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:14.036954 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:14.037714 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:59:14.037724 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:59:14.039407 (+ 1683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:59:14.041005 (+ 1598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:59:14.041011 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:59:14.041014 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:59:14.041432 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:59:14.041675 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:59:14.041676 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:59:14.041676 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:59:14.041829 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:59:14.042169 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:59:14.042176 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:59:14.042358 (+ 182us) client_negotiation.cc:769] Sending connection context
0926 03:59:14.042485 (+ 127us) client_negotiation.cc:240] Negotiation successful
0926 03:59:14.042677 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":97,"threads_started":1}
I20260926 03:59:14.043625 10950 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43501
I20260926 03:59:14.043723 10950 heartbeater.cc:464] Registering TS with master...
I20260926 03:59:14.043913 10950 heartbeater.cc:511] Master 127.0.98.62:43501 requested a full tablet report, sending...
I20260926 03:59:14.044343 10625 ts_manager.cc:194] Registered new tserver with Master: 983d20573bf54a119bda135144b9a4af (127.0.98.2:34011)
I20260926 03:59:14.044947 10625 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:39539
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 03:59:14.064479 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43501
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:59:14.145329 10955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:59:14.145546 10955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:59:14.145570 10955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:59:14.146684 10955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:59:14.146720 10955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:59:14.146746 10955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 03:59:14.148113 10955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43501
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:59:14.148293 10955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:59:14.148470 10955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:59:14.153448 10963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:59:14.153532 10955 server_base.cc:1034] running on GCE node
W20260926 03:59:14.153448 10961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:59:14.153482 10960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:59:14.153838 10955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:59:14.154088 10955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:59:14.155224 10955 hybrid_clock.cc:648] HybridClock initialized: now 1790395154155209 us; error 38 us; skew 500 ppm
Sep 26 03:59:14 dist-test-slave-nhpw krb5kdc[10580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395154, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:59:14.157866 10955 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 03:59:14.158231 10955 webserver.cc:460] Webserver started at http://127.0.98.3:34293/ using document root <none> and password file <none>
I20260926 03:59:14.158411 10955 fs_manager.cc:359] Metadata directory not provided
I20260926 03:59:14.158450 10955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:59:14.158536 10955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 03:59:14.159519 10955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/minicluster-data/ts-2/data/instance:
uuid: "a900f1bfeb7f4b0e8268505cf64abb88"
format_stamp: "Formatted at 2026-09-26 03:59:14 on dist-test-slave-nhpw"
server_key: "fe39446bc2e4c4193d4339b8b59dc7cb"
server_key_iv: "5a4af8cbfa261ba84d573b1bf007decd"
server_key_version: "encryptionkey@0"
I20260926 03:59:14.159812 10955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a900f1bfeb7f4b0e8268505cf64abb88"
format_stamp: "Formatted at 2026-09-26 03:59:14 on dist-test-slave-nhpw"
server_key: "fe39446bc2e4c4193d4339b8b59dc7cb"
server_key_iv: "5a4af8cbfa261ba84d573b1bf007decd"
server_key_version: "encryptionkey@0"
I20260926 03:59:14.161099 10955 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 03:59:14.161829 10970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:59:14.161985 10955 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 03:59:14.162066 10955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/minicluster-data/ts-2/wal
uuid: "a900f1bfeb7f4b0e8268505cf64abb88"
format_stamp: "Formatted at 2026-09-26 03:59:14 on dist-test-slave-nhpw"
server_key: "fe39446bc2e4c4193d4339b8b59dc7cb"
server_key_iv: "5a4af8cbfa261ba84d573b1bf007decd"
server_key_version: "encryptionkey@0"
I20260926 03:59:14.162117 10955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:59:14.172183 10955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:59:14.174081 10955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:59:14.174243 10955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:59:14.174475 10955 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:59:14.174774 10955 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:59:14.174826 10955 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:59:14.174868 10955 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:59:14.174917 10955 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:59:14.180812 10955 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37587
I20260926 03:59:14.180855 11083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37587 every 8 connection(s)
I20260926 03:59:14.181214 10955 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
Sep 26 03:59:14 dist-test-slave-nhpw krb5kdc[10580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395154, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 03:59:14.189360 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 10955
I20260926 03:59:14.189502 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/minicluster-data/ts-2/wal/instance
I20260926 03:59:14.189730 392 external_mini_cluster.cc:1428] Setting key d4136e41e8ceee33176913929fb7ede1
I20260926 03:59:14.190697 10682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:14.181959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47889 (local address 127.0.98.62:43501)
0926 03:59:14.182002 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 03:59:14.182004 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:59:14.182270 (+ 266us) server_negotiation.cc:407] Connection header received
0926 03:59:14.182725 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:59:14.182726 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:59:14.182736 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:59:14.182789 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:59:14.183725 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:14.184221 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:14.185256 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:14.185449 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:14.185509 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:14.187602 (+ 2093us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:59:14.187604 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:59:14.187604 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:59:14.187610 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:59:14.188974 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:59:14.189505 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:59:14.189506 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:59:14.189507 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:59:14.189577 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:59:14.189889 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:59:14.189890 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:59:14.189891 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:59:14.190069 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:59:14.190146 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 03:59:14.190505 (+ 359us) server_negotiation.cc:299] Negotiation successful
0926 03:59:14.190589 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 03:59:14.190893 11086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:14.181954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43501 (local address 127.0.98.3:47889)
0926 03:59:14.182109 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 03:59:14.182144 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 03:59:14.182644 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:59:14.182841 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:59:14.182845 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:59:14.183426 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:59:14.183617 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:14.183622 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:14.184310 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:14.184310 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:14.185175 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:14.185183 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:14.185527 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:14.185528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:14.185625 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:14.186073 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:59:14.186081 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:59:14.187506 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:59:14.189064 (+ 1558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:59:14.189070 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:59:14.189072 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:59:14.189422 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:59:14.189663 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:59:14.189664 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:59:14.189664 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:59:14.189804 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:59:14.190149 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:59:14.190155 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:59:14.190337 (+ 182us) client_negotiation.cc:769] Sending connection context
0926 03:59:14.190476 (+ 139us) client_negotiation.cc:240] Negotiation successful
0926 03:59:14.190638 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":92,"threads_started":1}
I20260926 03:59:14.191320 11084 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43501
I20260926 03:59:14.191455 11084 heartbeater.cc:464] Registering TS with master...
I20260926 03:59:14.191637 11084 heartbeater.cc:511] Master 127.0.98.62:43501 requested a full tablet report, sending...
I20260926 03:59:14.192003 10625 ts_manager.cc:194] Registered new tserver with Master: a900f1bfeb7f4b0e8268505cf64abb88 (127.0.98.3:37587)
I20260926 03:59:14.192360 10625 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47889
I20260926 03:59:14.201686 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:59:14.208901 10682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:14.203041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43634 (local address 127.0.98.62:43501)
0926 03:59:14.203120 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 03:59:14.203122 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:59:14.203177 (+ 55us) server_negotiation.cc:407] Connection header received
0926 03:59:14.203239 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:59:14.203240 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:59:14.203249 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:59:14.203290 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:59:14.203848 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:14.204322 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:14.205095 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:14.205273 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:14.205328 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:14.206450 (+ 1122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:59:14.206452 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:59:14.206452 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:59:14.206457 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:59:14.207831 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:59:14.208159 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:59:14.208160 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:59:14.208160 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:59:14.208208 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:59:14.208386 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:59:14.208386 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:59:14.208386 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:59:14.208515 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:59:14.208588 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 03:59:14.208764 (+ 176us) server_negotiation.cc:299] Negotiation successful
0926 03:59:14.208822 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 03:59:14.210405 10625 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43634:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:59:14.212231 10625 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:59:14.218683 11093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:14.214677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33624 (local address 127.0.98.2:34011)
0926 03:59:14.214883 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 03:59:14.214888 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:59:14.214898 (+ 10us) server_negotiation.cc:407] Connection header received
0926 03:59:14.214937 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:59:14.214939 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:59:14.214955 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:59:14.215234 (+ 279us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:59:14.215312 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:14.216037 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:14.217567 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:14.218280 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:14.218357 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:14.218416 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 03:59:14.218565 (+ 149us) server_negotiation.cc:299] Negotiation successful
0926 03:59:14.218602 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":83,"threads_started":1}
I20260926 03:59:14.218669 11092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:14.214560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34011 (local address 127.0.0.1:33624)
0926 03:59:14.214695 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 03:59:14.214726 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 03:59:14.214807 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:59:14.215076 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:59:14.215080 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:59:14.215083 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:59:14.215150 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:14.215152 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:14.216132 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:14.216133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:14.217480 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:14.217483 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:14.218372 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:14.218373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:14.218452 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:14.218468 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 03:59:14.218541 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 03:59:14.218571 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":5,"thread_start_us":73,"threads_started":1}
I20260926 03:59:14.218957 11097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:14.215124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43090 (local address 127.0.98.1:41103)
0926 03:59:14.215283 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 03:59:14.215287 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:59:14.215297 (+ 10us) server_negotiation.cc:407] Connection header received
0926 03:59:14.215334 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:59:14.215336 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:59:14.215353 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:59:14.215470 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:59:14.215569 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:14.216353 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:14.217731 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:14.218298 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:14.218738 (+ 440us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:14.218816 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 03:59:14.218846 (+ 30us) server_negotiation.cc:299] Negotiation successful
0926 03:59:14.218889 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":95,"threads_started":1}
I20260926 03:59:14.218673 11094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:14.214778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41103 (local address 127.0.0.1:43090)
0926 03:59:14.214851 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0926 03:59:14.214861 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 03:59:14.214907 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:59:14.215445 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:59:14.215446 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:59:14.215448 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:59:14.215506 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:14.215508 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:14.216484 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:14.216485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:14.217574 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:14.217576 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:14.218417 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:14.218418 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:14.218478 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:14.218481 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 03:59:14.218515 (+ 34us) client_negotiation.cc:240] Negotiation successful
0926 03:59:14.218569 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60,"thread_start_us":31,"threads_started":1}
I20260926 03:59:14.219657 11095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:14.214904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37587 (local address 127.0.0.1:37276)
0926 03:59:14.215013 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0926 03:59:14.215023 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 03:59:14.215064 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:59:14.216341 (+ 1277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:59:14.216342 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:59:14.216344 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:59:14.216421 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:14.216423 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:14.217312 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:14.217313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:14.218366 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:14.218369 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:14.219433 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:14.219434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:14.219494 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:14.219497 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 03:59:14.219559 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 03:59:14.219596 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":65,"threads_started":1}
I20260926 03:59:14.219802 11096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:14.215010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37276 (local address 127.0.98.3:37587)
0926 03:59:14.216182 (+ 1172us) server_negotiation.cc:206] Beginning negotiation
0926 03:59:14.216201 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 03:59:14.216208 (+ 7us) server_negotiation.cc:407] Connection header received
0926 03:59:14.216239 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:59:14.216240 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:59:14.216253 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:59:14.216344 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:59:14.216482 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:14.217207 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:14.218466 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:14.219166 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:14.219244 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:14.219301 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 03:59:14.219588 (+ 287us) server_negotiation.cc:299] Negotiation successful
0926 03:59:14.219633 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1140,"thread_start_us":97,"threads_started":1}
I20260926 03:59:14.219841 10884 tablet_service.cc:1467] Processing CreateTablet for tablet 674ca2d8e70b4cefabbfd2fa1032807c (DEFAULT_TABLE table=test-table [id=3fda93e6fc1d418e938b2695c9cca3f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:59:14.219856 10750 tablet_service.cc:1467] Processing CreateTablet for tablet 674ca2d8e70b4cefabbfd2fa1032807c (DEFAULT_TABLE table=test-table [id=3fda93e6fc1d418e938b2695c9cca3f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:59:14.220178 10884 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 674ca2d8e70b4cefabbfd2fa1032807c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:59:14.220191 10750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 674ca2d8e70b4cefabbfd2fa1032807c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:59:14.222623 11099 tablet_bootstrap.cc:492] T 674ca2d8e70b4cefabbfd2fa1032807c P 1958d3dc405249bc9721c7dab0cff947: Bootstrap starting.
I20260926 03:59:14.223315 11099 tablet_bootstrap.cc:654] T 674ca2d8e70b4cefabbfd2fa1032807c P 1958d3dc405249bc9721c7dab0cff947: Neither blocks nor log segments found. Creating new log.
I20260926 03:59:14.223464 11018 tablet_service.cc:1467] Processing CreateTablet for tablet 674ca2d8e70b4cefabbfd2fa1032807c (DEFAULT_TABLE table=test-table [id=3fda93e6fc1d418e938b2695c9cca3f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:59:14.223764 11018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 674ca2d8e70b4cefabbfd2fa1032807c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:59:14.223893 11099 log.cc:824] T 674ca2d8e70b4cefabbfd2fa1032807c P 1958d3dc405249bc9721c7dab0cff947: Log is configured to *not* fsync() on all Append() calls
I20260926 03:59:14.224246 11098 tablet_bootstrap.cc:492] T 674ca2d8e70b4cefabbfd2fa1032807c P 983d20573bf54a119bda135144b9a4af: Bootstrap starting.
I20260926 03:59:14.224787 11099 tablet_bootstrap.cc:492] T 674ca2d8e70b4cefabbfd2fa1032807c P 1958d3dc405249bc9721c7dab0cff947: No bootstrap required, opened a new log
I20260926 03:59:14.224900 11098 tablet_bootstrap.cc:654] T 674ca2d8e70b4cefabbfd2fa1032807c P 983d20573bf54a119bda135144b9a4af: Neither blocks nor log segments found. Creating new log.
I20260926 03:59:14.224938 11099 ts_tablet_manager.cc:1397] T 674ca2d8e70b4cefabbfd2fa1032807c P 1958d3dc405249bc9721c7dab0cff947: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 03:59:14.225478 11098 log.cc:824] T 674ca2d8e70b4cefabbfd2fa1032807c P 983d20573bf54a119bda135144b9a4af: Log is configured to *not* fsync() on all Append() calls
I20260926 03:59:14.226543 11102 tablet_bootstrap.cc:492] T 674ca2d8e70b4cefabbfd2fa1032807c P a900f1bfeb7f4b0e8268505cf64abb88: Bootstrap starting.
I20260926 03:59:14.226859 11099 raft_consensus.cc:348] T 674ca2d8e70b4cefabbfd2fa1032807c P 1958d3dc405249bc9721c7dab0cff947 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983d20573bf54a119bda135144b9a4af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34011 } } peers { permanent_uuid: "1958d3dc405249bc9721c7dab0cff947" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41103 } } peers { permanent_uuid: "a900f1bfeb7f4b0e8268505cf64abb88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37587 } }
I20260926 03:59:14.226999 11099 raft_consensus.cc:374] T 674ca2d8e70b4cefabbfd2fa1032807c P 1958d3dc405249bc9721c7dab0cff947 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:59:14.227022 11099 raft_consensus.cc:729] T 674ca2d8e70b4cefabbfd2fa1032807c P 1958d3dc405249bc9721c7dab0cff947 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1958d3dc405249bc9721c7dab0cff947, State: Initialized, Role: FOLLOWER
I20260926 03:59:14.227144 11102 tablet_bootstrap.cc:654] T 674ca2d8e70b4cefabbfd2fa1032807c P a900f1bfeb7f4b0e8268505cf64abb88: Neither blocks nor log segments found. Creating new log.
I20260926 03:59:14.227144 11099 consensus_queue.cc:260] T 674ca2d8e70b4cefabbfd2fa1032807c P 1958d3dc405249bc9721c7dab0cff947 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983d20573bf54a119bda135144b9a4af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34011 } } peers { permanent_uuid: "1958d3dc405249bc9721c7dab0cff947" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41103 } } peers { permanent_uuid: "a900f1bfeb7f4b0e8268505cf64abb88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37587 } }
I20260926 03:59:14.227422 11099 ts_tablet_manager.cc:1428] T 674ca2d8e70b4cefabbfd2fa1032807c P 1958d3dc405249bc9721c7dab0cff947: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 03:59:14.227504 10816 heartbeater.cc:503] Master 127.0.98.62:43501 was elected leader, sending a full tablet report...
I20260926 03:59:14.227756 11102 log.cc:824] T 674ca2d8e70b4cefabbfd2fa1032807c P a900f1bfeb7f4b0e8268505cf64abb88: Log is configured to *not* fsync() on all Append() calls
I20260926 03:59:14.228034 11098 tablet_bootstrap.cc:492] T 674ca2d8e70b4cefabbfd2fa1032807c P 983d20573bf54a119bda135144b9a4af: No bootstrap required, opened a new log
I20260926 03:59:14.228117 11098 ts_tablet_manager.cc:1397] T 674ca2d8e70b4cefabbfd2fa1032807c P 983d20573bf54a119bda135144b9a4af: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260926 03:59:14.228899 11102 tablet_bootstrap.cc:492] T 674ca2d8e70b4cefabbfd2fa1032807c P a900f1bfeb7f4b0e8268505cf64abb88: No bootstrap required, opened a new log
I20260926 03:59:14.228953 11102 ts_tablet_manager.cc:1397] T 674ca2d8e70b4cefabbfd2fa1032807c P a900f1bfeb7f4b0e8268505cf64abb88: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 03:59:14.230120 11098 raft_consensus.cc:348] T 674ca2d8e70b4cefabbfd2fa1032807c P 983d20573bf54a119bda135144b9a4af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983d20573bf54a119bda135144b9a4af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34011 } } peers { permanent_uuid: "1958d3dc405249bc9721c7dab0cff947" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41103 } } peers { permanent_uuid: "a900f1bfeb7f4b0e8268505cf64abb88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37587 } }
I20260926 03:59:14.230279 11098 raft_consensus.cc:374] T 674ca2d8e70b4cefabbfd2fa1032807c P 983d20573bf54a119bda135144b9a4af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:59:14.230321 11098 raft_consensus.cc:729] T 674ca2d8e70b4cefabbfd2fa1032807c P 983d20573bf54a119bda135144b9a4af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 983d20573bf54a119bda135144b9a4af, State: Initialized, Role: FOLLOWER
I20260926 03:59:14.230386 11102 raft_consensus.cc:348] T 674ca2d8e70b4cefabbfd2fa1032807c P a900f1bfeb7f4b0e8268505cf64abb88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983d20573bf54a119bda135144b9a4af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34011 } } peers { permanent_uuid: "1958d3dc405249bc9721c7dab0cff947" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41103 } } peers { permanent_uuid: "a900f1bfeb7f4b0e8268505cf64abb88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37587 } }
I20260926 03:59:14.230443 11098 consensus_queue.cc:260] T 674ca2d8e70b4cefabbfd2fa1032807c P 983d20573bf54a119bda135144b9a4af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983d20573bf54a119bda135144b9a4af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34011 } } peers { permanent_uuid: "1958d3dc405249bc9721c7dab0cff947" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41103 } } peers { permanent_uuid: "a900f1bfeb7f4b0e8268505cf64abb88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37587 } }
I20260926 03:59:14.230512 11102 raft_consensus.cc:374] T 674ca2d8e70b4cefabbfd2fa1032807c P a900f1bfeb7f4b0e8268505cf64abb88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:59:14.230535 11102 raft_consensus.cc:729] T 674ca2d8e70b4cefabbfd2fa1032807c P a900f1bfeb7f4b0e8268505cf64abb88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a900f1bfeb7f4b0e8268505cf64abb88, State: Initialized, Role: FOLLOWER
I20260926 03:59:14.230643 11102 consensus_queue.cc:260] T 674ca2d8e70b4cefabbfd2fa1032807c P a900f1bfeb7f4b0e8268505cf64abb88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983d20573bf54a119bda135144b9a4af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34011 } } peers { permanent_uuid: "1958d3dc405249bc9721c7dab0cff947" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41103 } } peers { permanent_uuid: "a900f1bfeb7f4b0e8268505cf64abb88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37587 } }
I20260926 03:59:14.230728 11098 ts_tablet_manager.cc:1428] T 674ca2d8e70b4cefabbfd2fa1032807c P 983d20573bf54a119bda135144b9a4af: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 03:59:14.230787 10950 heartbeater.cc:503] Master 127.0.98.62:43501 was elected leader, sending a full tablet report...
I20260926 03:59:14.230921 11102 ts_tablet_manager.cc:1428] T 674ca2d8e70b4cefabbfd2fa1032807c P a900f1bfeb7f4b0e8268505cf64abb88: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 03:59:14.230953 11084 heartbeater.cc:503] Master 127.0.98.62:43501 was elected leader, sending a full tablet report...
W20260926 03:59:14.282779 10951 tablet.cc:2369] T 674ca2d8e70b4cefabbfd2fa1032807c P 983d20573bf54a119bda135144b9a4af: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:59:14.298209 11103 raft_consensus.cc:482] T 674ca2d8e70b4cefabbfd2fa1032807c P 1958d3dc405249bc9721c7dab0cff947 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:59:14.298328 11103 raft_consensus.cc:504] T 674ca2d8e70b4cefabbfd2fa1032807c P 1958d3dc405249bc9721c7dab0cff947 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983d20573bf54a119bda135144b9a4af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34011 } } peers { permanent_uuid: "1958d3dc405249bc9721c7dab0cff947" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41103 } } peers { permanent_uuid: "a900f1bfeb7f4b0e8268505cf64abb88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37587 } }
I20260926 03:59:14.298655 11103 leader_election.cc:288] T 674ca2d8e70b4cefabbfd2fa1032807c P 1958d3dc405249bc9721c7dab0cff947 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 983d20573bf54a119bda135144b9a4af (127.0.98.2:34011), a900f1bfeb7f4b0e8268505cf64abb88 (127.0.98.3:37587)
I20260926 03:59:14.301640 11093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:14.298943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50289 (local address 127.0.98.2:34011)
0926 03:59:14.299038 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0926 03:59:14.299040 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:59:14.299050 (+ 10us) server_negotiation.cc:407] Connection header received
0926 03:59:14.299080 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:59:14.299081 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:59:14.299094 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:59:14.299175 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:59:14.299390 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:14.299921 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:14.300871 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:14.301279 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:14.301335 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:14.301343 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 03:59:14.301523 (+ 180us) server_negotiation.cc:299] Negotiation successful
0926 03:59:14.301567 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260926 03:59:14.301687 10818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:14.298853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34011 (local address 127.0.98.1:50289)
0926 03:59:14.298920 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0926 03:59:14.298933 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 03:59:14.298978 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:59:14.299177 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:59:14.299178 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:59:14.299179 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:59:14.299274 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:14.299275 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:14.300009 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:14.300010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:14.300794 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:14.300796 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:14.301362 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:14.301364 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:14.301446 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:14.301450 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 03:59:14.301503 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 03:59:14.301569 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 03:59:14.301731 11096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:14.299059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58709 (local address 127.0.98.3:37587)
0926 03:59:14.299112 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 03:59:14.299114 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:59:14.299230 (+ 116us) server_negotiation.cc:407] Connection header received
0926 03:59:14.299283 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:59:14.299284 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:59:14.299298 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:59:14.299353 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:59:14.299519 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:14.299989 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:14.300971 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:14.301357 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:14.301429 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:14.301438 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 03:59:14.301615 (+ 177us) server_negotiation.cc:299] Negotiation successful
0926 03:59:14.301656 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 03:59:14.301817 11107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:14.298980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37587 (local address 127.0.98.1:58709)
0926 03:59:14.299151 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 03:59:14.299161 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 03:59:14.299223 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:59:14.299371 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:59:14.299372 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:59:14.299374 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:59:14.299448 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:14.299450 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:14.300070 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:14.300071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:14.300890 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:14.300892 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:14.301447 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:14.301448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:14.301531 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:14.301535 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 03:59:14.301588 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 03:59:14.301624 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":103,"threads_started":1}
I20260926 03:59:14.302156 11038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674ca2d8e70b4cefabbfd2fa1032807c" candidate_uuid: "1958d3dc405249bc9721c7dab0cff947" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a900f1bfeb7f4b0e8268505cf64abb88" is_pre_election: true
I20260926 03:59:14.302160 10904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674ca2d8e70b4cefabbfd2fa1032807c" candidate_uuid: "1958d3dc405249bc9721c7dab0cff947" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "983d20573bf54a119bda135144b9a4af" is_pre_election: true
I20260926 03:59:14.302312 10904 raft_consensus.cc:2454] T 674ca2d8e70b4cefabbfd2fa1032807c P 983d20573bf54a119bda135144b9a4af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1958d3dc405249bc9721c7dab0cff947 in term 0.
I20260926 03:59:14.302318 11038 raft_consensus.cc:2454] T 674ca2d8e70b4cefabbfd2fa1032807c P a900f1bfeb7f4b0e8268505cf64abb88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1958d3dc405249bc9721c7dab0cff947 in term 0.
I20260926 03:59:14.302523 10705 leader_election.cc:302] T 674ca2d8e70b4cefabbfd2fa1032807c P 1958d3dc405249bc9721c7dab0cff947 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1958d3dc405249bc9721c7dab0cff947, 983d20573bf54a119bda135144b9a4af; no voters:
I20260926 03:59:14.302685 11103 raft_consensus.cc:2781] T 674ca2d8e70b4cefabbfd2fa1032807c P 1958d3dc405249bc9721c7dab0cff947 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:59:14.302757 11103 raft_consensus.cc:482] T 674ca2d8e70b4cefabbfd2fa1032807c P 1958d3dc405249bc9721c7dab0cff947 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:59:14.302775 11103 raft_consensus.cc:3037] T 674ca2d8e70b4cefabbfd2fa1032807c P 1958d3dc405249bc9721c7dab0cff947 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:59:14.303540 11103 raft_consensus.cc:504] T 674ca2d8e70b4cefabbfd2fa1032807c P 1958d3dc405249bc9721c7dab0cff947 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983d20573bf54a119bda135144b9a4af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34011 } } peers { permanent_uuid: "1958d3dc405249bc9721c7dab0cff947" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41103 } } peers { permanent_uuid: "a900f1bfeb7f4b0e8268505cf64abb88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37587 } }
I20260926 03:59:14.303671 11103 leader_election.cc:288] T 674ca2d8e70b4cefabbfd2fa1032807c P 1958d3dc405249bc9721c7dab0cff947 [CANDIDATE]: Term 1 election: Requested vote from peers 983d20573bf54a119bda135144b9a4af (127.0.98.2:34011), a900f1bfeb7f4b0e8268505cf64abb88 (127.0.98.3:37587)
I20260926 03:59:14.303841 10904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674ca2d8e70b4cefabbfd2fa1032807c" candidate_uuid: "1958d3dc405249bc9721c7dab0cff947" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "983d20573bf54a119bda135144b9a4af"
I20260926 03:59:14.303912 10904 raft_consensus.cc:3037] T 674ca2d8e70b4cefabbfd2fa1032807c P 983d20573bf54a119bda135144b9a4af [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:59:14.303869 11038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674ca2d8e70b4cefabbfd2fa1032807c" candidate_uuid: "1958d3dc405249bc9721c7dab0cff947" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a900f1bfeb7f4b0e8268505cf64abb88"
I20260926 03:59:14.303970 11038 raft_consensus.cc:3037] T 674ca2d8e70b4cefabbfd2fa1032807c P a900f1bfeb7f4b0e8268505cf64abb88 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:59:14.304625 10904 raft_consensus.cc:2454] T 674ca2d8e70b4cefabbfd2fa1032807c P 983d20573bf54a119bda135144b9a4af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1958d3dc405249bc9721c7dab0cff947 in term 1.
I20260926 03:59:14.304767 11038 raft_consensus.cc:2454] T 674ca2d8e70b4cefabbfd2fa1032807c P a900f1bfeb7f4b0e8268505cf64abb88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1958d3dc405249bc9721c7dab0cff947 in term 1.
I20260926 03:59:14.304852 10705 leader_election.cc:302] T 674ca2d8e70b4cefabbfd2fa1032807c P 1958d3dc405249bc9721c7dab0cff947 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1958d3dc405249bc9721c7dab0cff947, 983d20573bf54a119bda135144b9a4af; no voters:
I20260926 03:59:14.304940 11103 raft_consensus.cc:2781] T 674ca2d8e70b4cefabbfd2fa1032807c P 1958d3dc405249bc9721c7dab0cff947 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:59:14.305089 11103 raft_consensus.cc:686] T 674ca2d8e70b4cefabbfd2fa1032807c P 1958d3dc405249bc9721c7dab0cff947 [term 1 LEADER]: Becoming Leader. State: Replica: 1958d3dc405249bc9721c7dab0cff947, State: Running, Role: LEADER
I20260926 03:59:14.305162 11103 consensus_queue.cc:237] T 674ca2d8e70b4cefabbfd2fa1032807c P 1958d3dc405249bc9721c7dab0cff947 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983d20573bf54a119bda135144b9a4af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34011 } } peers { permanent_uuid: "1958d3dc405249bc9721c7dab0cff947" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41103 } } peers { permanent_uuid: "a900f1bfeb7f4b0e8268505cf64abb88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37587 } }
I20260926 03:59:14.306003 10625 catalog_manager.cc:5495] T 674ca2d8e70b4cefabbfd2fa1032807c P 1958d3dc405249bc9721c7dab0cff947 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1958d3dc405249bc9721c7dab0cff947 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "1958d3dc405249bc9721c7dab0cff947" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "983d20573bf54a119bda135144b9a4af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34011 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1958d3dc405249bc9721c7dab0cff947" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41103 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a900f1bfeb7f4b0e8268505cf64abb88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37587 } health_report { overall_health: UNKNOWN } } }
I20260926 03:59:14.312973 11097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:14.310603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43094 (local address 127.0.98.1:41103)
0926 03:59:14.310658 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 03:59:14.310662 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 03:59:14.310675 (+ 13us) server_negotiation.cc:407] Connection header received
0926 03:59:14.310731 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:59:14.310732 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:59:14.310747 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:59:14.310806 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:59:14.311004 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:14.311497 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:14.312269 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:14.312514 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:14.312566 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:14.312732 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:59:14.312820 (+ 88us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:59:14.312878 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 03:59:14.312892 (+ 14us) server_negotiation.cc:299] Negotiation successful
0926 03:59:14.312908 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 03:59:14.314690 11038 raft_consensus.cc:1261] T 674ca2d8e70b4cefabbfd2fa1032807c P a900f1bfeb7f4b0e8268505cf64abb88 [term 1 FOLLOWER]: Refusing update from remote peer 1958d3dc405249bc9721c7dab0cff947: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:59:14.314994 10904 raft_consensus.cc:1261] T 674ca2d8e70b4cefabbfd2fa1032807c P 983d20573bf54a119bda135144b9a4af [term 1 FOLLOWER]: Refusing update from remote peer 1958d3dc405249bc9721c7dab0cff947: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:59:14.315068 11108 consensus_queue.cc:1035] T 674ca2d8e70b4cefabbfd2fa1032807c P 1958d3dc405249bc9721c7dab0cff947 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a900f1bfeb7f4b0e8268505cf64abb88" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37587 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:59:14.315228 11103 consensus_queue.cc:1035] T 674ca2d8e70b4cefabbfd2fa1032807c P 1958d3dc405249bc9721c7dab0cff947 [LEADER]: Connected to new peer: Peer: permanent_uuid: "983d20573bf54a119bda135144b9a4af" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:59:14.317256 11111 mvcc.cc:205] Tried to move back new op lower bound from 7333458552070119424 to 7333458552034480128. Current Snapshot: MvccSnapshot[applied={T|T < 7333458552070119424}]
I20260926 03:59:14.317382 11113 mvcc.cc:205] Tried to move back new op lower bound from 7333458552070119424 to 7333458552034480128. Current Snapshot: MvccSnapshot[applied={T|T < 7333458552070119424}]
I20260926 03:59:14.318228 11112 mvcc.cc:205] Tried to move back new op lower bound from 7333458552070119424 to 7333458552034480128. Current Snapshot: MvccSnapshot[applied={T|T < 7333458552070119424}]
I20260926 03:59:14.319734 10624 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43634:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:59:14.319832 10624 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43634:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:59:14.320524 10624 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1423b97378254cd5ae147f3fb9db4c51: Sending DeleteTablet for 3 replicas of tablet 674ca2d8e70b4cefabbfd2fa1032807c
I20260926 03:59:14.320842 10884 tablet_service.cc:1514] Processing DeleteTablet for tablet 674ca2d8e70b4cefabbfd2fa1032807c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:59:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33624
I20260926 03:59:14.320923 11018 tablet_service.cc:1514] Processing DeleteTablet for tablet 674ca2d8e70b4cefabbfd2fa1032807c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:59:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37276
I20260926 03:59:14.321034 10750 tablet_service.cc:1514] Processing DeleteTablet for tablet 674ca2d8e70b4cefabbfd2fa1032807c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:59:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43090
I20260926 03:59:14.321192 11121 tablet_replica.cc:330] T 674ca2d8e70b4cefabbfd2fa1032807c P a900f1bfeb7f4b0e8268505cf64abb88: stopping tablet replica
I20260926 03:59:14.321273 11122 tablet_replica.cc:330] T 674ca2d8e70b4cefabbfd2fa1032807c P 1958d3dc405249bc9721c7dab0cff947: stopping tablet replica
I20260926 03:59:14.321370 11122 raft_consensus.cc:2229] T 674ca2d8e70b4cefabbfd2fa1032807c P 1958d3dc405249bc9721c7dab0cff947 [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:59:14.321437 11120 tablet_replica.cc:330] T 674ca2d8e70b4cefabbfd2fa1032807c P 983d20573bf54a119bda135144b9a4af: stopping tablet replica
I20260926 03:59:14.321463 11122 raft_consensus.cc:2258] T 674ca2d8e70b4cefabbfd2fa1032807c P 1958d3dc405249bc9721c7dab0cff947 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:59:14.321498 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 10687
I20260926 03:59:14.321525 11120 raft_consensus.cc:2229] T 674ca2d8e70b4cefabbfd2fa1032807c P 983d20573bf54a119bda135144b9a4af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:59:14.321621 11120 raft_consensus.cc:2258] T 674ca2d8e70b4cefabbfd2fa1032807c P 983d20573bf54a119bda135144b9a4af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:59:14.323633 11120 ts_tablet_manager.cc:1905] T 674ca2d8e70b4cefabbfd2fa1032807c P 983d20573bf54a119bda135144b9a4af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:59:14.325265 11121 raft_consensus.cc:2229] T 674ca2d8e70b4cefabbfd2fa1032807c P a900f1bfeb7f4b0e8268505cf64abb88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:59:14.325426 11121 raft_consensus.cc:2258] T 674ca2d8e70b4cefabbfd2fa1032807c P a900f1bfeb7f4b0e8268505cf64abb88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:59:14.325644 11120 ts_tablet_manager.cc:1918] T 674ca2d8e70b4cefabbfd2fa1032807c P 983d20573bf54a119bda135144b9a4af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:59:14.325860 11120 log.cc:1196] T 674ca2d8e70b4cefabbfd2fa1032807c P 983d20573bf54a119bda135144b9a4af: Deleting WAL directory at /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/minicluster-data/ts-1/wal/wals/674ca2d8e70b4cefabbfd2fa1032807c
I20260926 03:59:14.325945 11121 ts_tablet_manager.cc:1905] T 674ca2d8e70b4cefabbfd2fa1032807c P a900f1bfeb7f4b0e8268505cf64abb88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:59:14.326131 11120 ts_tablet_manager.cc:1939] T 674ca2d8e70b4cefabbfd2fa1032807c P 983d20573bf54a119bda135144b9a4af: Deleting consensus metadata
I20260926 03:59:14.326583 10611 catalog_manager.cc:4841] TS 983d20573bf54a119bda135144b9a4af (127.0.98.2:34011): tablet 674ca2d8e70b4cefabbfd2fa1032807c (table test-table [id=3fda93e6fc1d418e938b2695c9cca3f5]) successfully deleted
I20260926 03:59:14.327383 11121 ts_tablet_manager.cc:1918] T 674ca2d8e70b4cefabbfd2fa1032807c P a900f1bfeb7f4b0e8268505cf64abb88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260926 03:59:14.327383 10611 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41103 (error 108)
I20260926 03:59:14.327456 11121 log.cc:1196] T 674ca2d8e70b4cefabbfd2fa1032807c P a900f1bfeb7f4b0e8268505cf64abb88: Deleting WAL directory at /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790395095474643-392-0/minicluster-data/ts-2/wal/wals/674ca2d8e70b4cefabbfd2fa1032807c
I20260926 03:59:14.327620 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 10821
I20260926 03:59:14.327732 11121 ts_tablet_manager.cc:1939] T 674ca2d8e70b4cefabbfd2fa1032807c P a900f1bfeb7f4b0e8268505cf64abb88: Deleting consensus metadata
I20260926 03:59:14.328207 11095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:14.327779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41103 (local address 127.0.0.1:43096)
0926 03:59:14.328127 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0926 03:59:14.328167 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327}
W20260926 03:59:14.328361 10611 catalog_manager.cc:4584] TS 1958d3dc405249bc9721c7dab0cff947 (127.0.98.1:41103): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 674ca2d8e70b4cefabbfd2fa1032807c: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41103: connect: Connection refused (error 111)
I20260926 03:59:14.332481 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 10955
I20260926 03:59:14.336822 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 10593
I20260926 03:59:14.336988 11095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:14.336862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37587 (local address 127.0.0.1:37284)
0926 03:59:14.336927 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0926 03:59:14.336953 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.3:37587: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
2026-09-26T03:59:14Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1918 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 03:59:14 dist-test-slave-nhpw krb5kdc[11126](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 03:59:14 dist-test-slave-nhpw krb5kdc[11126](info): set up 2 sockets
Sep 26 03:59:14 dist-test-slave-nhpw krb5kdc[11126](info): commencing operation
krb5kdc: starting...
W20260926 03:59:15.385546 392 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/krb5kdc/test-user.keytab.
Sep 26 03:59:15 dist-test-slave-nhpw krb5kdc[11126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395155, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T03:59:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T03:59:15Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260926 03:59:15.555372 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task7pf3XC/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/postgres -l logfile start
2026-09-26 03:59:18.103 UTC [11156] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 03:59:18.103 UTC [11156] LOG: listening on IPv4 address "127.0.98.20", port 58307
2026-09-26 03:59:18.107 UTC [11156] LOG: listening on Unix socket "/tmp/.s.PGSQL.58307"
2026-09-26 03:59:18.128 UTC [11159] LOG: database system was shut down at 2026-09-26 03:59:16 UTC
2026-09-26 03:59:18.132 UTC [11156] LOG: database system is ready to accept connections
I20260926 03:59:19.081497 392 mini_postgres.cc:96] Postgres bound to 58307
2026-09-26 03:59:19.087 UTC [11167] FATAL: database "slave" does not exist
127.0.98.20:58307 - accepting connections
I20260926 03:59:19.087870 392 mini_ranger.cc:161] Starting Ranger...
I20260926 03:59:19.104040 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 03:59:19.640810 392 mini_ranger.cc:87] Created ranger Postgres database
I20260926 03:59:19.640880 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/ranger-admin
2026-09-26 03:59:19,912 [I] DB FLAVOR :POSTGRES
2026-09-26 03:59:19,912 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 03:59:19,912 [I] Checking connection..
2026-09-26 03:59:19,913 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7pf3XC/build/release/bin/postgresql.jar:/tmp/dist-test-task7pf3XC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58307/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 03:59:20,156 [I] Checking connection passed.
2026-09-26 03:59:20,156 [I] --------- Verifying version history table ---------
2026-09-26 03:59:20,157 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7pf3XC/build/release/bin/postgresql.jar:/tmp/dist-test-task7pf3XC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58307/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 03:59:20,410 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 03:59:20,410 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7pf3XC/build/release/bin/postgresql.jar:/tmp/dist-test-task7pf3XC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58307/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 03:59:20,669 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 03:59:20,669 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 03:59:20,669 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7pf3XC/build/release/bin/postgresql.jar:/tmp/dist-test-task7pf3XC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58307/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task7pf3XC/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 03:59:20.922 UTC [11231] WARNING: there is no transaction in progress
2026-09-26 03:59:20,928 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 03:59:20,928 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7pf3XC/build/release/bin/postgresql.jar:/tmp/dist-test-task7pf3XC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58307/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 03:59:21,191 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 03:59:21,192 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 03:59:21,192 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7pf3XC/build/release/bin/postgresql.jar:/tmp/dist-test-task7pf3XC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58307/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 03:59:21,448 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7pf3XC/build/release/bin/postgresql.jar:/tmp/dist-test-task7pf3XC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58307/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 03:59:21,698 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7pf3XC/build/release/bin/postgresql.jar:/tmp/dist-test-task7pf3XC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58307/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-nhpw.c.gcp-upstream.internal','N') ;"
2026-09-26 03:59:21,916 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7pf3XC/build/release/bin/postgresql.jar:/tmp/dist-test-task7pf3XC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58307/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 03:59:22,158 [I] Table x_portal_user does not exist in database ranger
2026-09-26 03:59:22,159 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7pf3XC/build/release/bin/postgresql.jar:/tmp/dist-test-task7pf3XC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58307/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 03:59:22,404 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 03:59:22,404 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7pf3XC/build/release/bin/postgresql.jar:/tmp/dist-test-task7pf3XC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58307/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 03:59:22,644 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7pf3XC/build/release/bin/postgresql.jar:/tmp/dist-test-task7pf3XC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58307/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 03:59:22,889 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 03:59:22,889 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7pf3XC/build/release/bin/postgresql.jar:/tmp/dist-test-task7pf3XC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58307/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task7pf3XC/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 03:59:23.888 UTC [11339] WARNING: there is no transaction in progress
2026-09-26 03:59:23.905 UTC [11339] WARNING: there is no transaction in progress
2026-09-26 03:59:23.922 UTC [11339] WARNING: there is no transaction in progress
2026-09-26 03:59:23.939 UTC [11339] WARNING: there is no transaction in progress
2026-09-26 03:59:23.957 UTC [11339] WARNING: there is no transaction in progress
2026-09-26 03:59:23.976 UTC [11339] WARNING: there is no transaction in progress
2026-09-26 03:59:24.045 UTC [11339] WARNING: there is no transaction in progress
2026-09-26 03:59:24.053 UTC [11339] WARNING: there is no transaction in progress
2026-09-26 03:59:24.071 UTC [11339] WARNING: there is no transaction in progress
2026-09-26 03:59:24.082 UTC [11339] WARNING: there is no transaction in progress
2026-09-26 03:59:24.094 UTC [11339] WARNING: there is no transaction in progress
2026-09-26 03:59:24.111 UTC [11339] WARNING: there is no transaction in progress
2026-09-26 03:59:24.126 UTC [11339] WARNING: there is no transaction in progress
2026-09-26 03:59:24.139 UTC [11339] WARNING: there is no transaction in progress
2026-09-26 03:59:24.160 UTC [11339] WARNING: there is no transaction in progress
2026-09-26 03:59:24.169 UTC [11339] WARNING: there is no transaction in progress
2026-09-26 03:59:25,074 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 03:59:25,075 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7pf3XC/build/release/bin/postgresql.jar:/tmp/dist-test-task7pf3XC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58307/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 03:59:25,303 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 03:59:25,303 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7pf3XC/build/release/bin/postgresql.jar:/tmp/dist-test-task7pf3XC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58307/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 03:59:25,541 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 03:59:25,541 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7pf3XC/build/release/bin/postgresql.jar:/tmp/dist-test-task7pf3XC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58307/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 03:59:25,798 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 03:59:25,799 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7pf3XC/build/release/bin/postgresql.jar:/tmp/dist-test-task7pf3XC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58307/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 03:59:26,031 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7pf3XC/build/release/bin/postgresql.jar:/tmp/dist-test-task7pf3XC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58307/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 03:59:26,273 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7pf3XC/build/release/bin/postgresql.jar:/tmp/dist-test-task7pf3XC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58307/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-nhpw.c.gcp-upstream.internal';"
2026-09-26 03:59:26,491 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 03:59:26,492 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7pf3XC/build/release/bin/postgresql.jar:/tmp/dist-test-task7pf3XC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58307/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 03:59:26,728 [I] DB_PATCHES have already been applied
I20260926 03:59:26.733666 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/ranger-admin:/tmp/dist-test-task7pf3XC/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task7pf3XC/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task7pf3XC/build/release/bin/postgresql.jar:/tmp/dist-test-task7pf3XC/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 03:59:26.733726 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260926 03:59:26.736717 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:59133
Sep 26, 2026 3:59:27 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 3:59:27 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 3:59:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 3:59:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 3:59:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 3:59:27 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 3:59:27 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 03:59:27.847901 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.760s user 0.000s sys 0.009s
Sep 26, 2026 3:59:47 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 3:59:47 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 26, 2026 3:59:47 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 3:59:47 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 3:59:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 3:59:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 3:59:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 3:59:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 3:59:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 3:59:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 3:59:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 3:59:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 3:59:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 3:59:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 3:59:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 3:59:47 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 3:59:47 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 3:59:48 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 03:59:48.953220 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 03:59:49 dist-test-slave-nhpw krb5kdc[11126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395189, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 03:59:49.043744 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 03:59:49.053589 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 03:59:50.120577 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 03:59:50.120711 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/ranger-kms
I20260926 03:59:50.120743 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:58307
2026-09-26 03:59:50,300 [I] DB FLAVOR :POSTGRES
2026-09-26 03:59:50,300 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 03:59:50,301 [I] Checking connection
2026-09-26 03:59:50,301 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7pf3XC/build/release/bin/postgresql.jar:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58307/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 03:59:50,571 [I] connection success
2026-09-26 03:59:50,571 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 03:59:50,571 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 03:59:50,571 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7pf3XC/build/release/bin/postgresql.jar:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58307/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 03:59:50,821 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 03:59:50,821 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 03:59:50,821 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 03:59:50,821 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7pf3XC/build/release/bin/postgresql.jar:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:58307/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 03:59:51,092 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 03:59:51.564889 392 mini_ranger_kms.cc:310] Created kms service
I20260926 03:59:51.729952 392 mini_ranger_kms.cc:326] Created kudu user
I20260926 03:59:51.773408 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 03:59:51 dist-test-slave-nhpw krb5kdc[11126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395189, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 03:59:52.161667 392 mini_ranger_kms.cc:384] Added ranger policy
I20260926 03:59:52.161785 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/ranger-kms:/tmp/dist-test-task7pf3XC/build/release/bin/postgresql.jar:/tmp/dist-test-task7pf3XC/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task7pf3XC/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 03:59:52.161810 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260926 03:59:52.165571 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11519
I20260926 03:59:52.165707 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:35107
Sep 26, 2026 3:59:53 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 3:59:53 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 3:59:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 3:59:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 3:59:53 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 3:59:53 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 03:59:54.452775 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.409s user 0.008s sys 0.154s
I20260926 03:59:54.452883 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 03:59:54.452962 392 mini_ranger_kms.cc:401] 127.0.98.20:35107/kms/v1/keys
Sep 26 03:59:54 dist-test-slave-nhpw krb5kdc[11126](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790395194, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 03:59:55 dist-test-slave-nhpw krb5kdc[11126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395194, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 26 03:59:55 dist-test-slave-nhpw krb5kdc[11126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395194, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 26 03:59:57 dist-test-slave-nhpw krb5kdc[11126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 03:59:57.547801 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33259
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33259
--ranger_config_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:35107/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:59:57.644407 11570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:59:57.644649 11570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:59:57.644733 11570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:59:57.645876 11570 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 03:59:57.645946 11570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:59:57.645970 11570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:59:57.645998 11570 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 03:59:57.646030 11570 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 03:59:57.647581 11570 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:35107/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33259
--ranger_config_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11570
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:59:57.647850 11570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:59:57.648083 11570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:59:57.653503 11578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:59:57.653544 11576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:59:57.653594 11570 server_base.cc:1034] running on GCE node
W20260926 03:59:57.653545 11575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:59:57.654016 11570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:59:57.654316 11570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:59:57.655507 11570 hybrid_clock.cc:648] HybridClock initialized: now 1790395197655487 us; error 41 us; skew 500 ppm
Sep 26 03:59:57 dist-test-slave-nhpw krb5kdc[11126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395197, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:59:57.658243 11570 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 03:59:57.658733 11570 webserver.cc:460] Webserver started at http://127.0.98.62:36019/ using document root <none> and password file <none>
I20260926 03:59:57.658964 11570 fs_manager.cc:359] Metadata directory not provided
I20260926 03:59:57.659005 11570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:59:57.659161 11570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 03:59:57 dist-test-slave-nhpw krb5kdc[11126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395197, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 03:59:57.712908 11570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/master-0/data/instance:
uuid: "4754d31874144ebf91ef26140b8674da"
format_stamp: "Formatted at 2026-09-26 03:59:57 on dist-test-slave-nhpw"
server_key: "2c8185c8dff15840a282ab0ebb96b1e9"
server_key_iv: "d934e20b73cb3ce09afb3b9d59906c87"
server_key_version: "kuduclusterkey@0"
I20260926 03:59:57.713387 11570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/master-0/wal/instance:
uuid: "4754d31874144ebf91ef26140b8674da"
format_stamp: "Formatted at 2026-09-26 03:59:57 on dist-test-slave-nhpw"
server_key: "2c8185c8dff15840a282ab0ebb96b1e9"
server_key_iv: "d934e20b73cb3ce09afb3b9d59906c87"
server_key_version: "kuduclusterkey@0"
I20260926 03:59:57.715178 11570 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260926 03:59:57.727138 11585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:59:57.727309 11570 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 03:59:57.727404 11570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/master-0/wal
uuid: "4754d31874144ebf91ef26140b8674da"
format_stamp: "Formatted at 2026-09-26 03:59:57 on dist-test-slave-nhpw"
server_key: "2c8185c8dff15840a282ab0ebb96b1e9"
server_key_iv: "d934e20b73cb3ce09afb3b9d59906c87"
server_key_version: "kuduclusterkey@0"
I20260926 03:59:57.727509 11570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:59:57.761921 11570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:59:57.764189 11570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:59:57.764369 11570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:59:57.769373 11570 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33259
I20260926 03:59:57.769397 11637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33259 every 8 connection(s)
I20260926 03:59:57.769781 11570 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
I20260926 03:59:57.770440 11638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:59:57.772626 11638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4754d31874144ebf91ef26140b8674da: Bootstrap starting.
I20260926 03:59:57.773279 11638 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4754d31874144ebf91ef26140b8674da: Neither blocks nor log segments found. Creating new log.
I20260926 03:59:57.773551 11638 log.cc:824] T 00000000000000000000000000000000 P 4754d31874144ebf91ef26140b8674da: Log is configured to *not* fsync() on all Append() calls
I20260926 03:59:57.774462 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 11570
I20260926 03:59:57.774564 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/master-0/wal/instance
I20260926 03:59:57.774549 11638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4754d31874144ebf91ef26140b8674da: No bootstrap required, opened a new log
I20260926 03:59:57.774765 392 external_mini_cluster.cc:1428] Setting key 06abafe2f5db726a88a8812491bc9bc3
I20260926 03:59:57.777596 11638 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4754d31874144ebf91ef26140b8674da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4754d31874144ebf91ef26140b8674da" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33259 } }
I20260926 03:59:57.777768 11638 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4754d31874144ebf91ef26140b8674da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:59:57.777832 11638 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4754d31874144ebf91ef26140b8674da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4754d31874144ebf91ef26140b8674da, State: Initialized, Role: FOLLOWER
I20260926 03:59:57.777995 11638 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4754d31874144ebf91ef26140b8674da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4754d31874144ebf91ef26140b8674da" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33259 } }
I20260926 03:59:57.778090 11638 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4754d31874144ebf91ef26140b8674da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 03:59:57.778144 11638 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4754d31874144ebf91ef26140b8674da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 03:59:57.778216 11638 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4754d31874144ebf91ef26140b8674da [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:59:57.778853 11638 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4754d31874144ebf91ef26140b8674da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4754d31874144ebf91ef26140b8674da" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33259 } }
I20260926 03:59:57.778963 11638 leader_election.cc:302] T 00000000000000000000000000000000 P 4754d31874144ebf91ef26140b8674da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4754d31874144ebf91ef26140b8674da; no voters:
I20260926 03:59:57.779168 11638 leader_election.cc:288] T 00000000000000000000000000000000 P 4754d31874144ebf91ef26140b8674da [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 03:59:57.779210 11643 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4754d31874144ebf91ef26140b8674da [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:59:57.779435 11643 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4754d31874144ebf91ef26140b8674da [term 1 LEADER]: Becoming Leader. State: Replica: 4754d31874144ebf91ef26140b8674da, State: Running, Role: LEADER
I20260926 03:59:57.779558 11643 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4754d31874144ebf91ef26140b8674da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4754d31874144ebf91ef26140b8674da" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33259 } }
I20260926 03:59:57.779606 11638 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4754d31874144ebf91ef26140b8674da [sys.catalog]: configured and running, proceeding with master startup.
I20260926 03:59:57.781345 11643 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4754d31874144ebf91ef26140b8674da [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4754d31874144ebf91ef26140b8674da. Latest consensus state: current_term: 1 leader_uuid: "4754d31874144ebf91ef26140b8674da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4754d31874144ebf91ef26140b8674da" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33259 } } }
I20260926 03:59:57.781492 11643 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4754d31874144ebf91ef26140b8674da [sys.catalog]: This master's current role is: LEADER
I20260926 03:59:57.781634 11638 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 03:59:57.781711 11657 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 03:59:57.782477 11644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4754d31874144ebf91ef26140b8674da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4754d31874144ebf91ef26140b8674da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4754d31874144ebf91ef26140b8674da" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33259 } } }
I20260926 03:59:57.782675 11644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4754d31874144ebf91ef26140b8674da [sys.catalog]: This master's current role is: LEADER
I20260926 03:59:57.783336 11657 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 03:59:57 dist-test-slave-nhpw krb5kdc[11126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 03:59:57.788492 11657 catalog_manager.cc:1347] Generated new cluster ID: e578d587b3c04deca24ee36b570fbca5
I20260926 03:59:57.788662 11657 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 03:59:57.790649 11642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:57.775311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49856 (local address 127.0.98.62:33259)
0926 03:59:57.775480 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 03:59:57.775489 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 03:59:57.775519 (+ 30us) server_negotiation.cc:407] Connection header received
0926 03:59:57.776076 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:59:57.776085 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:59:57.776148 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:59:57.776300 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:59:57.777213 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:57.778613 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:57.779756 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:57.780370 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:57.780475 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:57.785176 (+ 4701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:59:57.785222 (+ 46us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:59:57.785223 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:59:57.785308 (+ 85us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:59:57.788700 (+ 3392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:59:57.789109 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:59:57.789112 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:59:57.789113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:59:57.789211 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:59:57.789397 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:59:57.789398 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:59:57.789398 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:59:57.789678 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:59:57.790228 (+ 550us) server_negotiation.cc:1035] Waiting for connection context
0926 03:59:57.790296 (+ 68us) server_negotiation.cc:299] Negotiation successful
0926 03:59:57.790449 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":68,"threads_started":1}
I20260926 03:59:57.824524 11657 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 03:59:57.825394 11657 catalog_manager.cc:1504] Loading token signing keys...
I20260926 03:59:57.839501 11657 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4754d31874144ebf91ef26140b8674da: Generated new TSK 0
I20260926 03:59:57.839818 11657 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 03:59:58.508678 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:35107/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33259
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:59:58.595330 11683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:59:58.595557 11683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:59:58.595597 11683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:59:58.596840 11683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:59:58.596900 11683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:59:58.596946 11683 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 03:59:58.598416 11683 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:35107/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11683
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:59:58.598677 11683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:59:58.598917 11683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:59:58.604002 11688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:59:58.604017 11689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:59:58.604086 11691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:59:58.604676 11683 server_base.cc:1034] running on GCE node
I20260926 03:59:58.604894 11683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:59:58.605139 11683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:59:58.606318 11683 hybrid_clock.cc:648] HybridClock initialized: now 1790395198606295 us; error 48 us; skew 500 ppm
Sep 26 03:59:58 dist-test-slave-nhpw krb5kdc[11126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395198, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:59:58.609079 11683 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 03:59:58.609587 11683 webserver.cc:460] Webserver started at http://127.0.98.1:37095/ using document root <none> and password file <none>
I20260926 03:59:58.609824 11683 fs_manager.cc:359] Metadata directory not provided
I20260926 03:59:58.609889 11683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:59:58.610023 11683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 03:59:58 dist-test-slave-nhpw krb5kdc[11126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395198, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 03:59:58.629458 11683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/ts-0/data/instance:
uuid: "e5434739be5a4d488b286f8c0307eff9"
format_stamp: "Formatted at 2026-09-26 03:59:58 on dist-test-slave-nhpw"
server_key: "c912b21e6f178808f0dc6026e0f2a1e3"
server_key_iv: "73a0918bbd495d690f3ccf7f20b8fe7e"
server_key_version: "kuduclusterkey@0"
I20260926 03:59:58.629997 11683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e5434739be5a4d488b286f8c0307eff9"
format_stamp: "Formatted at 2026-09-26 03:59:58 on dist-test-slave-nhpw"
server_key: "c912b21e6f178808f0dc6026e0f2a1e3"
server_key_iv: "73a0918bbd495d690f3ccf7f20b8fe7e"
server_key_version: "kuduclusterkey@0"
I20260926 03:59:58.635408 11683 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.000s
I20260926 03:59:58.650946 11700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:59:58.651161 11683 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 03:59:58.651247 11683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/ts-0/wal
uuid: "e5434739be5a4d488b286f8c0307eff9"
format_stamp: "Formatted at 2026-09-26 03:59:58 on dist-test-slave-nhpw"
server_key: "c912b21e6f178808f0dc6026e0f2a1e3"
server_key_iv: "73a0918bbd495d690f3ccf7f20b8fe7e"
server_key_version: "kuduclusterkey@0"
I20260926 03:59:58.651338 11683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:59:58.666620 11683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:59:58.669219 11683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:59:58.669399 11683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:59:58.669613 11683 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:59:58.669970 11683 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:59:58.670017 11683 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:59:58.670047 11683 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:59:58.670107 11683 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:59:58.678452 11683 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39887
I20260926 03:59:58.678560 11813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39887 every 8 connection(s)
I20260926 03:59:58.678920 11683 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
I20260926 03:59:58.687505 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 11683
Sep 26 03:59:58 dist-test-slave-nhpw krb5kdc[11126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395198, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 03:59:58.687608 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/ts-0/wal/instance
I20260926 03:59:58.687793 392 external_mini_cluster.cc:1428] Setting key e3389834453da222daf64a0ccad88bc9
I20260926 03:59:58.691767 11817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:58.679846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44719 (local address 127.0.98.62:33259)
0926 03:59:58.681681 (+ 1835us) server_negotiation.cc:206] Beginning negotiation
0926 03:59:58.681684 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 03:59:58.681692 (+ 8us) server_negotiation.cc:407] Connection header received
0926 03:59:58.681718 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:59:58.681719 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:59:58.681744 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:59:58.681821 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:59:58.682384 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:58.683578 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:58.685040 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:58.685356 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:58.685437 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:58.688194 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:59:58.688195 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:59:58.688196 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:59:58.688203 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:59:58.690116 (+ 1913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:59:58.690609 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:59:58.690611 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:59:58.690611 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:59:58.690681 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:59:58.690978 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:59:58.690979 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:59:58.690979 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:59:58.691122 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:59:58.691197 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 03:59:58.691550 (+ 353us) server_negotiation.cc:299] Negotiation successful
0926 03:59:58.691643 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1803,"thread_start_us":93,"threads_started":1}
I20260926 03:59:58.691874 11816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:58.679808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33259 (local address 127.0.98.1:44719)
0926 03:59:58.680755 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0926 03:59:58.680805 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 03:59:58.681490 (+ 685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:59:58.681896 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:59:58.681900 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:59:58.682078 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:59:58.682280 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:58.682284 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:58.683668 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:58.683669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:58.684941 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:58.684944 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:58.685460 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:58.685461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:58.685606 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:58.686242 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:59:58.686252 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:59:58.687910 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:59:58.690205 (+ 2295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:59:58.690211 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:59:58.690214 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:59:58.690513 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:59:58.690759 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:59:58.690760 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:59:58.690760 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:59:58.690896 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:59:58.691201 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:59:58.691206 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:59:58.691382 (+ 176us) client_negotiation.cc:769] Sending connection context
0926 03:59:58.691501 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 03:59:58.691651 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":892,"thread_start_us":82,"threads_started":1}
I20260926 03:59:58.692732 11814 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33259
I20260926 03:59:58.692840 11814 heartbeater.cc:464] Registering TS with master...
I20260926 03:59:58.693020 11814 heartbeater.cc:511] Master 127.0.98.62:33259 requested a full tablet report, sending...
I20260926 03:59:58.693526 11602 ts_manager.cc:194] Registered new tserver with Master: e5434739be5a4d488b286f8c0307eff9 (127.0.98.1:39887)
I20260926 03:59:58.694399 11602 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:44719
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 03:59:58.728770 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:35107/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33259
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:59:58.829366 11820 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:59:58.829581 11820 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:59:58.829620 11820 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:59:58.830929 11820 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:59:58.830986 11820 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:59:58.831029 11820 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 03:59:58.832828 11820 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:35107/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11820
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:59:58.833074 11820 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:59:58.833334 11820 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Sep 26 03:59:58 dist-test-slave-nhpw krb5kdc[11126](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790395198, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 03:59:58.839020 11828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:59:58.839028 11830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:59:58.839452 11827 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:59:58.839470 11820 server_base.cc:1034] running on GCE node
I20260926 03:59:58.839897 11820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:59:58.840215 11820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:59:58.841416 11820 hybrid_clock.cc:648] HybridClock initialized: now 1790395198841391 us; error 33 us; skew 500 ppm
Sep 26 03:59:58 dist-test-slave-nhpw krb5kdc[11126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395198, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:59:58.844204 11820 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 03:59:58.844774 11820 webserver.cc:460] Webserver started at http://127.0.98.2:39175/ using document root <none> and password file <none>
I20260926 03:59:58.845008 11820 fs_manager.cc:359] Metadata directory not provided
I20260926 03:59:58.845077 11820 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:59:58.845223 11820 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 03:59:58 dist-test-slave-nhpw krb5kdc[11126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395198, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 03:59:58.862807 11820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/ts-1/data/instance:
uuid: "a666f62187f9490baca7bcd1bbd7a00b"
format_stamp: "Formatted at 2026-09-26 03:59:58 on dist-test-slave-nhpw"
server_key: "26f8c0d31014185f1f63f0f40aaf83fa"
server_key_iv: "6b3b130dccfdcec918d04d16fb53df75"
server_key_version: "kuduclusterkey@0"
I20260926 03:59:58.863397 11820 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a666f62187f9490baca7bcd1bbd7a00b"
format_stamp: "Formatted at 2026-09-26 03:59:58 on dist-test-slave-nhpw"
server_key: "26f8c0d31014185f1f63f0f40aaf83fa"
server_key_iv: "6b3b130dccfdcec918d04d16fb53df75"
server_key_version: "kuduclusterkey@0"
I20260926 03:59:58.865471 11820 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260926 03:59:58.878260 11837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:59:58.878505 11820 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s
I20260926 03:59:58.878587 11820 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/ts-1/wal
uuid: "a666f62187f9490baca7bcd1bbd7a00b"
format_stamp: "Formatted at 2026-09-26 03:59:58 on dist-test-slave-nhpw"
server_key: "26f8c0d31014185f1f63f0f40aaf83fa"
server_key_iv: "6b3b130dccfdcec918d04d16fb53df75"
server_key_version: "kuduclusterkey@0"
I20260926 03:59:58.878682 11820 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:59:58.925935 11820 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:59:58.928475 11820 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:59:58.928719 11820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:59:58.928983 11820 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:59:58.929378 11820 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:59:58.929466 11820 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:59:58.929548 11820 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:59:58.929617 11820 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:59:58.936565 11820 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38559
I20260926 03:59:58.937124 11820 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
I20260926 03:59:58.941087 11951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38559 every 8 connection(s)
Sep 26 03:59:58 dist-test-slave-nhpw krb5kdc[11126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395198, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 03:59:58.947149 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 11820
I20260926 03:59:58.947252 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/ts-1/wal/instance
I20260926 03:59:58.947454 392 external_mini_cluster.cc:1428] Setting key 0cd2eaf93a3e32753549dade2085a9d0
I20260926 03:59:58.951490 11817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:58.941767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36619 (local address 127.0.98.62:33259)
0926 03:59:58.941835 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 03:59:58.941838 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 03:59:58.942088 (+ 250us) server_negotiation.cc:407] Connection header received
0926 03:59:58.942618 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:59:58.942619 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:59:58.942631 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:59:58.942685 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:59:58.943255 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:58.943778 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:58.944845 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:58.945063 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:58.945154 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:58.947491 (+ 2337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:59:58.947492 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:59:58.947493 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:59:58.947502 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:59:58.949358 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:59:58.949890 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:59:58.949892 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:59:58.949892 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:59:58.950340 (+ 448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:59:58.950648 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:59:58.950649 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:59:58.950650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:59:58.950795 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:59:58.950840 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 03:59:58.951246 (+ 406us) server_negotiation.cc:299] Negotiation successful
0926 03:59:58.951347 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 03:59:58.952039 11954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:58.941763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33259 (local address 127.0.98.2:36619)
0926 03:59:58.941943 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 03:59:58.941985 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 03:59:58.942507 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:59:58.942759 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:59:58.942763 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:59:58.942957 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:59:58.943156 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:58.943160 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:58.943887 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:58.943888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:58.944749 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:58.944752 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:58.945163 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:58.945164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:58.945291 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:58.945866 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:59:58.945875 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:59:58.947369 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:59:58.949455 (+ 2086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:59:58.949461 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:59:58.949464 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:59:58.949792 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:59:58.950431 (+ 639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:59:58.950432 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:59:58.950433 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:59:58.950573 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:59:58.950889 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:59:58.950895 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:59:58.951077 (+ 182us) client_negotiation.cc:769] Sending connection context
0926 03:59:58.951589 (+ 512us) client_negotiation.cc:240] Negotiation successful
0926 03:59:58.951764 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":85,"threads_started":1}
I20260926 03:59:58.952705 11952 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33259
I20260926 03:59:58.952800 11952 heartbeater.cc:464] Registering TS with master...
I20260926 03:59:58.952991 11952 heartbeater.cc:511] Master 127.0.98.62:33259 requested a full tablet report, sending...
I20260926 03:59:58.953431 11602 ts_manager.cc:194] Registered new tserver with Master: a666f62187f9490baca7bcd1bbd7a00b (127.0.98.2:38559)
I20260926 03:59:58.953943 11602 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36619
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 03:59:58.975632 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:35107/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33259
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39901
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 03:59:59.098158 11957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 03:59:59.098433 11957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 03:59:59.098471 11957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 03:59:59.100203 11957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 03:59:59.100268 11957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 03:59:59.100310 11957 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 03:59:59.102485 11957 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39901
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:35107/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33259
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 03:59:59.102885 11957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 03:59:59.103157 11957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 03:59:59.109205 11962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:59:59.109514 11963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 03:59:59.109823 11965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 03:59:59.110723 11957 server_base.cc:1034] running on GCE node
I20260926 03:59:59.110961 11957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 03:59:59.111217 11957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 03:59:59.112399 11957 hybrid_clock.cc:648] HybridClock initialized: now 1790395199112383 us; error 32 us; skew 500 ppm
Sep 26 03:59:59 dist-test-slave-nhpw krb5kdc[11126](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395199, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 03:59:59.114935 11957 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 03:59:59.115370 11957 webserver.cc:460] Webserver started at http://127.0.98.3:42625/ using document root <none> and password file <none>
I20260926 03:59:59.115609 11957 fs_manager.cc:359] Metadata directory not provided
I20260926 03:59:59.115677 11957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 03:59:59.115832 11957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 03:59:59 dist-test-slave-nhpw krb5kdc[11126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395199, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 03:59:59.128788 11957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/ts-2/data/instance:
uuid: "93a00a04d3e54c79aeb555bfd2ada656"
format_stamp: "Formatted at 2026-09-26 03:59:59 on dist-test-slave-nhpw"
server_key: "2ad34ce2ced794e3b80369296704f69b"
server_key_iv: "58da8138584986c787703eb28ec4d164"
server_key_version: "kuduclusterkey@0"
I20260926 03:59:59.129396 11957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/ts-2/wal/instance:
uuid: "93a00a04d3e54c79aeb555bfd2ada656"
format_stamp: "Formatted at 2026-09-26 03:59:59 on dist-test-slave-nhpw"
server_key: "2ad34ce2ced794e3b80369296704f69b"
server_key_iv: "58da8138584986c787703eb28ec4d164"
server_key_version: "kuduclusterkey@0"
I20260926 03:59:59.137259 11957 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.000s sys 0.001s
I20260926 03:59:59.151950 11972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 03:59:59.152257 11957 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 03:59:59.152490 11957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/ts-2/wal
uuid: "93a00a04d3e54c79aeb555bfd2ada656"
format_stamp: "Formatted at 2026-09-26 03:59:59 on dist-test-slave-nhpw"
server_key: "2ad34ce2ced794e3b80369296704f69b"
server_key_iv: "58da8138584986c787703eb28ec4d164"
server_key_version: "kuduclusterkey@0"
I20260926 03:59:59.152694 11957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 03:59:59.170192 11957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 03:59:59.172670 11957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 03:59:59.172871 11957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 03:59:59.173151 11957 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 03:59:59.173519 11957 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 03:59:59.173568 11957 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 03:59:59.173600 11957 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 03:59:59.173663 11957 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 03:59:59.181963 11957 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45905
I20260926 03:59:59.182413 11957 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
I20260926 03:59:59.182943 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 11957
I20260926 03:59:59.183045 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/ts-2/wal/instance
I20260926 03:59:59.183218 392 external_mini_cluster.cc:1428] Setting key 00f966c8e4fdbec9922943034d2edcb1
I20260926 03:59:59.185448 12085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45905 every 8 connection(s)
Sep 26 03:59:59 dist-test-slave-nhpw krb5kdc[11126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395199, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 03:59:59.194729 11817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:59.184279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51473 (local address 127.0.98.62:33259)
0926 03:59:59.184333 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 03:59:59.184335 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:59:59.184345 (+ 10us) server_negotiation.cc:407] Connection header received
0926 03:59:59.185135 (+ 790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:59:59.185136 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:59:59.185148 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:59:59.185214 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:59:59.185960 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:59.186502 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:59.188028 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:59.188333 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:59.188384 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:59.191130 (+ 2746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:59:59.191131 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:59:59.191132 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:59:59.191140 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:59:59.193011 (+ 1871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:59:59.193529 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:59:59.193530 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:59:59.193530 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:59:59.193610 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:59:59.193934 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:59:59.193935 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:59:59.193935 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:59:59.194073 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:59:59.194119 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 03:59:59.194536 (+ 417us) server_negotiation.cc:299] Negotiation successful
0926 03:59:59.194634 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 03:59:59.195055 12088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:59.184012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33259 (local address 127.0.98.3:51473)
0926 03:59:59.184206 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 03:59:59.184258 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 03:59:59.185012 (+ 754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:59:59.185288 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:59:59.185293 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:59:59.185558 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0926 03:59:59.185857 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:59.185864 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:59.186608 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:59.186609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:59.187918 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:59.187921 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:59.188483 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:59.188484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:59.188664 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:59.189297 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 03:59:59.189308 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 03:59:59.191016 (+ 1708us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 03:59:59.193104 (+ 2088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:59:59.193112 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:59:59.193114 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:59:59.193423 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:59:59.193708 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 03:59:59.193708 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 03:59:59.193709 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 03:59:59.193847 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 03:59:59.194172 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 03:59:59.194179 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 03:59:59.194355 (+ 176us) client_negotiation.cc:769] Sending connection context
0926 03:59:59.194488 (+ 133us) client_negotiation.cc:240] Negotiation successful
0926 03:59:59.194665 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":85,"threads_started":1}
I20260926 03:59:59.195753 12086 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33259
I20260926 03:59:59.195858 12086 heartbeater.cc:464] Registering TS with master...
I20260926 03:59:59.196107 12086 heartbeater.cc:511] Master 127.0.98.62:33259 requested a full tablet report, sending...
I20260926 03:59:59.196537 11602 ts_manager.cc:194] Registered new tserver with Master: 93a00a04d3e54c79aeb555bfd2ada656 (127.0.98.3:45905)
I20260926 03:59:59.196978 11602 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:51473
I20260926 03:59:59.206446 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 03:59:59.216313 11817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:59.208086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49858 (local address 127.0.98.62:33259)
0926 03:59:59.208131 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 03:59:59.208133 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:59:59.208458 (+ 325us) server_negotiation.cc:407] Connection header received
0926 03:59:59.208568 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:59:59.208569 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:59:59.208580 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:59:59.208612 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 03:59:59.209426 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:59.210199 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:59.211303 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:59.211592 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:59.211680 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:59.213257 (+ 1577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 03:59:59.213258 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 03:59:59.213258 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 03:59:59.213266 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 03:59:59.215019 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:59:59.215428 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:59:59.215429 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:59:59.215429 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:59:59.215499 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 03:59:59.215736 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 03:59:59.215737 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 03:59:59.215738 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 03:59:59.215878 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 03:59:59.215977 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 03:59:59.216133 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 03:59:59.216220 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 03:59:59.218369 11602 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49858:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 03:59:59.220697 11602 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 03:59:59.229620 12099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:59.223793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39887 (local address 127.0.0.1:59058)
0926 03:59:59.224172 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0926 03:59:59.224209 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 03:59:59.224271 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:59:59.225018 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:59:59.225022 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:59:59.225024 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:59:59.225092 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:59.225094 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:59.226373 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:59.226374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:59.227978 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:59.227981 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:59.229326 (+ 1345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:59.229327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:59.229415 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:59.229428 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 03:59:59.229491 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 03:59:59.229540 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":69,"threads_started":1}
I20260926 03:59:59.229924 12101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:59.224034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59058 (local address 127.0.98.1:39887)
0926 03:59:59.224405 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0926 03:59:59.224410 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:59:59.224419 (+ 9us) server_negotiation.cc:407] Connection header received
0926 03:59:59.224832 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:59:59.224835 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:59:59.224854 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:59:59.224941 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:59:59.225168 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:59.226243 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:59.228076 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:59.229167 (+ 1091us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:59.229230 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:59.229283 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 03:59:59.229817 (+ 534us) server_negotiation.cc:299] Negotiation successful
0926 03:59:59.229859 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":50,"threads_started":1}
I20260926 03:59:59.230604 11748 tablet_service.cc:1467] Processing CreateTablet for tablet d57344652d534e21ab070ac069a86015 (DEFAULT_TABLE table=test-table [id=1b32a4cda64d490c9b09ff74319957b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:59:59.230968 11748 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d57344652d534e21ab070ac069a86015. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:59:59.236203 12100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:59.223799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45905 (local address 127.0.0.1:41424)
0926 03:59:59.224721 (+ 922us) negotiation.cc:107] Waiting for socket to connect
0926 03:59:59.224731 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 03:59:59.224782 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:59:59.229039 (+ 4257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:59:59.229040 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:59:59.229043 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:59:59.229093 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:59.229095 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:59.233873 (+ 4778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:59.233874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:59.234968 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:59.234971 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:59.235961 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:59.235962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:59.236029 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:59.236033 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 03:59:59.236094 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 03:59:59.236132 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":907,"thread_start_us":89,"threads_started":1}
I20260926 03:59:59.236433 12102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:59.224051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41424 (local address 127.0.98.3:45905)
0926 03:59:59.228800 (+ 4749us) server_negotiation.cc:206] Beginning negotiation
0926 03:59:59.228805 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:59:59.228816 (+ 11us) server_negotiation.cc:407] Connection header received
0926 03:59:59.228855 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:59:59.228857 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:59:59.228873 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:59:59.228955 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:59:59.229308 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:59.233795 (+ 4487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:59.235093 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:59.235860 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:59.235943 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:59.236017 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 03:59:59.236320 (+ 303us) server_negotiation.cc:299] Negotiation successful
0926 03:59:59.236360 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4708,"thread_start_us":85,"threads_started":1}
I20260926 03:59:59.237183 12020 tablet_service.cc:1467] Processing CreateTablet for tablet d57344652d534e21ab070ac069a86015 (DEFAULT_TABLE table=test-table [id=1b32a4cda64d490c9b09ff74319957b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:59:59.237528 12020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d57344652d534e21ab070ac069a86015. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:59:59.237874 12103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:59.224286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38559 (local address 127.0.0.1:60726)
0926 03:59:59.229194 (+ 4908us) negotiation.cc:107] Waiting for socket to connect
0926 03:59:59.229204 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 03:59:59.229240 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:59:59.231285 (+ 2045us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:59:59.231286 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:59:59.231289 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:59:59.231328 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:59.231330 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:59.232573 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:59.232574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:59.233684 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:59.233687 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:59.237633 (+ 3946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:59.237634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:59.237708 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:59.237714 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 03:59:59.237761 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 03:59:59.237799 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4892,"thread_start_us":74,"threads_started":1}
I20260926 03:59:59.238380 12104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:59.224503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60726 (local address 127.0.98.2:38559)
0926 03:59:59.230913 (+ 6410us) server_negotiation.cc:206] Beginning negotiation
0926 03:59:59.230918 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 03:59:59.230930 (+ 12us) server_negotiation.cc:407] Connection header received
0926 03:59:59.230969 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:59:59.230971 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:59:59.230985 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:59:59.231199 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:59:59.231413 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:59.232472 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:59.236076 (+ 3604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:59.237507 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:59.238154 (+ 647us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:59.238241 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 03:59:59.238266 (+ 25us) server_negotiation.cc:299] Negotiation successful
0926 03:59:59.238304 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6369,"thread_start_us":75,"threads_started":1}
I20260926 03:59:59.239078 11886 tablet_service.cc:1467] Processing CreateTablet for tablet d57344652d534e21ab070ac069a86015 (DEFAULT_TABLE table=test-table [id=1b32a4cda64d490c9b09ff74319957b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 03:59:59.239413 11886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d57344652d534e21ab070ac069a86015. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 03:59:59.240760 12105 tablet_bootstrap.cc:492] T d57344652d534e21ab070ac069a86015 P e5434739be5a4d488b286f8c0307eff9: Bootstrap starting.
I20260926 03:59:59.241356 12105 tablet_bootstrap.cc:654] T d57344652d534e21ab070ac069a86015 P e5434739be5a4d488b286f8c0307eff9: Neither blocks nor log segments found. Creating new log.
I20260926 03:59:59.241564 12105 log.cc:824] T d57344652d534e21ab070ac069a86015 P e5434739be5a4d488b286f8c0307eff9: Log is configured to *not* fsync() on all Append() calls
I20260926 03:59:59.242664 12108 tablet_bootstrap.cc:492] T d57344652d534e21ab070ac069a86015 P a666f62187f9490baca7bcd1bbd7a00b: Bootstrap starting.
I20260926 03:59:59.243150 12106 tablet_bootstrap.cc:492] T d57344652d534e21ab070ac069a86015 P 93a00a04d3e54c79aeb555bfd2ada656: Bootstrap starting.
I20260926 03:59:59.244006 12108 tablet_bootstrap.cc:654] T d57344652d534e21ab070ac069a86015 P a666f62187f9490baca7bcd1bbd7a00b: Neither blocks nor log segments found. Creating new log.
I20260926 03:59:59.244290 12108 log.cc:824] T d57344652d534e21ab070ac069a86015 P a666f62187f9490baca7bcd1bbd7a00b: Log is configured to *not* fsync() on all Append() calls
I20260926 03:59:59.244330 12106 tablet_bootstrap.cc:654] T d57344652d534e21ab070ac069a86015 P 93a00a04d3e54c79aeb555bfd2ada656: Neither blocks nor log segments found. Creating new log.
I20260926 03:59:59.244771 12106 log.cc:824] T d57344652d534e21ab070ac069a86015 P 93a00a04d3e54c79aeb555bfd2ada656: Log is configured to *not* fsync() on all Append() calls
I20260926 03:59:59.245151 12105 tablet_bootstrap.cc:492] T d57344652d534e21ab070ac069a86015 P e5434739be5a4d488b286f8c0307eff9: No bootstrap required, opened a new log
I20260926 03:59:59.245226 12105 ts_tablet_manager.cc:1397] T d57344652d534e21ab070ac069a86015 P e5434739be5a4d488b286f8c0307eff9: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260926 03:59:59.245911 12108 tablet_bootstrap.cc:492] T d57344652d534e21ab070ac069a86015 P a666f62187f9490baca7bcd1bbd7a00b: No bootstrap required, opened a new log
I20260926 03:59:59.246045 12108 ts_tablet_manager.cc:1397] T d57344652d534e21ab070ac069a86015 P a666f62187f9490baca7bcd1bbd7a00b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 03:59:59.247637 12105 raft_consensus.cc:348] T d57344652d534e21ab070ac069a86015 P e5434739be5a4d488b286f8c0307eff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5434739be5a4d488b286f8c0307eff9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39887 } } peers { permanent_uuid: "a666f62187f9490baca7bcd1bbd7a00b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38559 } } peers { permanent_uuid: "93a00a04d3e54c79aeb555bfd2ada656" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45905 } }
I20260926 03:59:59.247833 12105 raft_consensus.cc:374] T d57344652d534e21ab070ac069a86015 P e5434739be5a4d488b286f8c0307eff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:59:59.247870 12105 raft_consensus.cc:729] T d57344652d534e21ab070ac069a86015 P e5434739be5a4d488b286f8c0307eff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5434739be5a4d488b286f8c0307eff9, State: Initialized, Role: FOLLOWER
I20260926 03:59:59.248001 12105 consensus_queue.cc:260] T d57344652d534e21ab070ac069a86015 P e5434739be5a4d488b286f8c0307eff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5434739be5a4d488b286f8c0307eff9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39887 } } peers { permanent_uuid: "a666f62187f9490baca7bcd1bbd7a00b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38559 } } peers { permanent_uuid: "93a00a04d3e54c79aeb555bfd2ada656" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45905 } }
I20260926 03:59:59.248245 12105 ts_tablet_manager.cc:1428] T d57344652d534e21ab070ac069a86015 P e5434739be5a4d488b286f8c0307eff9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 03:59:59.248368 11814 heartbeater.cc:503] Master 127.0.98.62:33259 was elected leader, sending a full tablet report...
I20260926 03:59:59.248417 12108 raft_consensus.cc:348] T d57344652d534e21ab070ac069a86015 P a666f62187f9490baca7bcd1bbd7a00b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5434739be5a4d488b286f8c0307eff9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39887 } } peers { permanent_uuid: "a666f62187f9490baca7bcd1bbd7a00b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38559 } } peers { permanent_uuid: "93a00a04d3e54c79aeb555bfd2ada656" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45905 } }
I20260926 03:59:59.248610 12108 raft_consensus.cc:374] T d57344652d534e21ab070ac069a86015 P a666f62187f9490baca7bcd1bbd7a00b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:59:59.248695 12108 raft_consensus.cc:729] T d57344652d534e21ab070ac069a86015 P a666f62187f9490baca7bcd1bbd7a00b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a666f62187f9490baca7bcd1bbd7a00b, State: Initialized, Role: FOLLOWER
I20260926 03:59:59.248839 12108 consensus_queue.cc:260] T d57344652d534e21ab070ac069a86015 P a666f62187f9490baca7bcd1bbd7a00b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5434739be5a4d488b286f8c0307eff9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39887 } } peers { permanent_uuid: "a666f62187f9490baca7bcd1bbd7a00b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38559 } } peers { permanent_uuid: "93a00a04d3e54c79aeb555bfd2ada656" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45905 } }
I20260926 03:59:59.249083 12108 ts_tablet_manager.cc:1428] T d57344652d534e21ab070ac069a86015 P a666f62187f9490baca7bcd1bbd7a00b: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260926 03:59:59.249212 11952 heartbeater.cc:503] Master 127.0.98.62:33259 was elected leader, sending a full tablet report...
I20260926 03:59:59.253317 12106 tablet_bootstrap.cc:492] T d57344652d534e21ab070ac069a86015 P 93a00a04d3e54c79aeb555bfd2ada656: No bootstrap required, opened a new log
I20260926 03:59:59.253396 12106 ts_tablet_manager.cc:1397] T d57344652d534e21ab070ac069a86015 P 93a00a04d3e54c79aeb555bfd2ada656: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260926 03:59:59.255293 12106 raft_consensus.cc:348] T d57344652d534e21ab070ac069a86015 P 93a00a04d3e54c79aeb555bfd2ada656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5434739be5a4d488b286f8c0307eff9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39887 } } peers { permanent_uuid: "a666f62187f9490baca7bcd1bbd7a00b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38559 } } peers { permanent_uuid: "93a00a04d3e54c79aeb555bfd2ada656" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45905 } }
I20260926 03:59:59.255447 12106 raft_consensus.cc:374] T d57344652d534e21ab070ac069a86015 P 93a00a04d3e54c79aeb555bfd2ada656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 03:59:59.255475 12106 raft_consensus.cc:729] T d57344652d534e21ab070ac069a86015 P 93a00a04d3e54c79aeb555bfd2ada656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93a00a04d3e54c79aeb555bfd2ada656, State: Initialized, Role: FOLLOWER
I20260926 03:59:59.255574 12106 consensus_queue.cc:260] T d57344652d534e21ab070ac069a86015 P 93a00a04d3e54c79aeb555bfd2ada656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5434739be5a4d488b286f8c0307eff9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39887 } } peers { permanent_uuid: "a666f62187f9490baca7bcd1bbd7a00b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38559 } } peers { permanent_uuid: "93a00a04d3e54c79aeb555bfd2ada656" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45905 } }
I20260926 03:59:59.255810 12106 ts_tablet_manager.cc:1428] T d57344652d534e21ab070ac069a86015 P 93a00a04d3e54c79aeb555bfd2ada656: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 03:59:59.256026 12086 heartbeater.cc:503] Master 127.0.98.62:33259 was elected leader, sending a full tablet report...
Sep 26 03:59:59 dist-test-slave-nhpw krb5kdc[11126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395198, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260926 03:59:59.430088 11815 tablet.cc:2369] T d57344652d534e21ab070ac069a86015 P e5434739be5a4d488b286f8c0307eff9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:59:59.435925 12087 tablet.cc:2369] T d57344652d534e21ab070ac069a86015 P 93a00a04d3e54c79aeb555bfd2ada656: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 03:59:59.441931 11953 tablet.cc:2369] T d57344652d534e21ab070ac069a86015 P a666f62187f9490baca7bcd1bbd7a00b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 03:59:59.542901 12112 raft_consensus.cc:482] T d57344652d534e21ab070ac069a86015 P a666f62187f9490baca7bcd1bbd7a00b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:59:59.543043 12112 raft_consensus.cc:504] T d57344652d534e21ab070ac069a86015 P a666f62187f9490baca7bcd1bbd7a00b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5434739be5a4d488b286f8c0307eff9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39887 } } peers { permanent_uuid: "a666f62187f9490baca7bcd1bbd7a00b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38559 } } peers { permanent_uuid: "93a00a04d3e54c79aeb555bfd2ada656" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45905 } }
I20260926 03:59:59.543438 12112 leader_election.cc:288] T d57344652d534e21ab070ac069a86015 P a666f62187f9490baca7bcd1bbd7a00b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e5434739be5a4d488b286f8c0307eff9 (127.0.98.1:39887), 93a00a04d3e54c79aeb555bfd2ada656 (127.0.98.3:45905)
I20260926 03:59:59.544123 12111 raft_consensus.cc:482] T d57344652d534e21ab070ac069a86015 P e5434739be5a4d488b286f8c0307eff9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 03:59:59.544225 12111 raft_consensus.cc:504] T d57344652d534e21ab070ac069a86015 P e5434739be5a4d488b286f8c0307eff9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5434739be5a4d488b286f8c0307eff9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39887 } } peers { permanent_uuid: "a666f62187f9490baca7bcd1bbd7a00b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38559 } } peers { permanent_uuid: "93a00a04d3e54c79aeb555bfd2ada656" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45905 } }
I20260926 03:59:59.544533 12111 leader_election.cc:288] T d57344652d534e21ab070ac069a86015 P e5434739be5a4d488b286f8c0307eff9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a666f62187f9490baca7bcd1bbd7a00b (127.0.98.2:38559), 93a00a04d3e54c79aeb555bfd2ada656 (127.0.98.3:45905)
Sep 26 03:59:59 dist-test-slave-nhpw krb5kdc[11126](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395198, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 03:59:59.552610 12117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:59.544819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38559 (local address 127.0.98.1:40211)
0926 03:59:59.548808 (+ 3989us) negotiation.cc:107] Waiting for socket to connect
0926 03:59:59.548831 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 03:59:59.548926 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:59:59.549131 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:59:59.549132 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:59:59.549134 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:59:59.549229 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:59.549232 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:59.550347 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:59.550348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:59.551562 (+ 1214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:59.551565 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:59.552338 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:59.552339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:59.552414 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:59.552419 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 03:59:59.552475 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 03:59:59.552516 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3967,"thread_start_us":81,"threads_started":1}
I20260926 03:59:59.552659 12104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:59.544767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40211 (local address 127.0.98.2:38559)
0926 03:59:59.544826 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 03:59:59.544829 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 03:59:59.548896 (+ 4067us) server_negotiation.cc:407] Connection header received
0926 03:59:59.549007 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:59:59.549008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:59:59.549021 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:59:59.549052 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:59:59.549546 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:59.550259 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:59.551670 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:59.552214 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:59.552262 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:59.552273 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 03:59:59.552514 (+ 241us) server_negotiation.cc:299] Negotiation successful
0926 03:59:59.552563 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":24}
I20260926 03:59:59.552975 11906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d57344652d534e21ab070ac069a86015" candidate_uuid: "e5434739be5a4d488b286f8c0307eff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a666f62187f9490baca7bcd1bbd7a00b" is_pre_election: true
I20260926 03:59:59.553097 11906 raft_consensus.cc:2454] T d57344652d534e21ab070ac069a86015 P a666f62187f9490baca7bcd1bbd7a00b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5434739be5a4d488b286f8c0307eff9 in term 0.
I20260926 03:59:59.553750 11702 leader_election.cc:302] T d57344652d534e21ab070ac069a86015 P e5434739be5a4d488b286f8c0307eff9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a666f62187f9490baca7bcd1bbd7a00b, e5434739be5a4d488b286f8c0307eff9; no voters:
I20260926 03:59:59.553917 12111 raft_consensus.cc:2781] T d57344652d534e21ab070ac069a86015 P e5434739be5a4d488b286f8c0307eff9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:59:59.553962 12111 raft_consensus.cc:482] T d57344652d534e21ab070ac069a86015 P e5434739be5a4d488b286f8c0307eff9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:59:59.553984 12111 raft_consensus.cc:3037] T d57344652d534e21ab070ac069a86015 P e5434739be5a4d488b286f8c0307eff9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:59:59.554555 12102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:59.543716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46147 (local address 127.0.98.3:45905)
0926 03:59:59.543766 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 03:59:59.543769 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 03:59:59.549730 (+ 5961us) server_negotiation.cc:407] Connection header received
0926 03:59:59.550018 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:59:59.550019 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:59:59.550028 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:59:59.550055 (+ 27us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:59:59.550247 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:59.550937 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:59.553614 (+ 2677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:59.554204 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:59.554266 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:59.554276 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 03:59:59.554454 (+ 178us) server_negotiation.cc:299] Negotiation successful
0926 03:59:59.554491 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 03:59:59.554893 12116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:59.543660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45905 (local address 127.0.98.2:46147)
0926 03:59:59.549657 (+ 5997us) negotiation.cc:107] Waiting for socket to connect
0926 03:59:59.549668 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 03:59:59.549928 (+ 260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:59:59.550111 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:59:59.550112 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:59:59.550113 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:59:59.550185 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:59.550187 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:59.552351 (+ 2164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:59.552352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:59.553512 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:59.553514 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:59.554307 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:59.554308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:59.554378 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:59.554381 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 03:59:59.554760 (+ 379us) client_negotiation.cc:240] Negotiation successful
0926 03:59:59.554801 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5982,"thread_start_us":78,"threads_started":1}
I20260926 03:59:59.554903 12111 raft_consensus.cc:504] T d57344652d534e21ab070ac069a86015 P e5434739be5a4d488b286f8c0307eff9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5434739be5a4d488b286f8c0307eff9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39887 } } peers { permanent_uuid: "a666f62187f9490baca7bcd1bbd7a00b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38559 } } peers { permanent_uuid: "93a00a04d3e54c79aeb555bfd2ada656" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45905 } }
I20260926 03:59:59.555071 12111 leader_election.cc:288] T d57344652d534e21ab070ac069a86015 P e5434739be5a4d488b286f8c0307eff9 [CANDIDATE]: Term 1 election: Requested vote from peers a666f62187f9490baca7bcd1bbd7a00b (127.0.98.2:38559), 93a00a04d3e54c79aeb555bfd2ada656 (127.0.98.3:45905)
I20260926 03:59:59.555299 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d57344652d534e21ab070ac069a86015" candidate_uuid: "a666f62187f9490baca7bcd1bbd7a00b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93a00a04d3e54c79aeb555bfd2ada656" is_pre_election: true
I20260926 03:59:59.555444 12040 raft_consensus.cc:2454] T d57344652d534e21ab070ac069a86015 P 93a00a04d3e54c79aeb555bfd2ada656 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a666f62187f9490baca7bcd1bbd7a00b in term 0.
I20260926 03:59:59.555678 11838 leader_election.cc:302] T d57344652d534e21ab070ac069a86015 P a666f62187f9490baca7bcd1bbd7a00b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93a00a04d3e54c79aeb555bfd2ada656, a666f62187f9490baca7bcd1bbd7a00b; no voters:
I20260926 03:59:59.555809 12112 raft_consensus.cc:2781] T d57344652d534e21ab070ac069a86015 P a666f62187f9490baca7bcd1bbd7a00b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 03:59:59.555851 12112 raft_consensus.cc:482] T d57344652d534e21ab070ac069a86015 P a666f62187f9490baca7bcd1bbd7a00b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 03:59:59.555873 12112 raft_consensus.cc:3037] T d57344652d534e21ab070ac069a86015 P a666f62187f9490baca7bcd1bbd7a00b [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:59:59.556785 12112 raft_consensus.cc:504] T d57344652d534e21ab070ac069a86015 P a666f62187f9490baca7bcd1bbd7a00b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5434739be5a4d488b286f8c0307eff9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39887 } } peers { permanent_uuid: "a666f62187f9490baca7bcd1bbd7a00b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38559 } } peers { permanent_uuid: "93a00a04d3e54c79aeb555bfd2ada656" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45905 } }
I20260926 03:59:59.556910 12112 leader_election.cc:288] T d57344652d534e21ab070ac069a86015 P a666f62187f9490baca7bcd1bbd7a00b [CANDIDATE]: Term 1 election: Requested vote from peers e5434739be5a4d488b286f8c0307eff9 (127.0.98.1:39887), 93a00a04d3e54c79aeb555bfd2ada656 (127.0.98.3:45905)
I20260926 03:59:59.557155 12101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:59.544016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50635 (local address 127.0.98.1:39887)
0926 03:59:59.544073 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 03:59:59.544075 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:59:59.553704 (+ 9629us) server_negotiation.cc:407] Connection header received
0926 03:59:59.553793 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:59:59.553794 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:59:59.553804 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:59:59.553832 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:59:59.553979 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:59.554631 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:59.556200 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:59.556786 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:59.556840 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:59.556847 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0926 03:59:59.557060 (+ 213us) server_negotiation.cc:299] Negotiation successful
0926 03:59:59.557098 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 03:59:59.557363 12115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:59.543859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39887 (local address 127.0.98.2:50635)
0926 03:59:59.553650 (+ 9791us) negotiation.cc:107] Waiting for socket to connect
0926 03:59:59.553660 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 03:59:59.553727 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:59:59.553884 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:59:59.553885 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:59:59.553886 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:59:59.553917 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:59.553918 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:59.554725 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:59.554726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:59.556078 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:59.556081 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:59.556906 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:59.556906 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:59.556974 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:59.556979 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 03:59:59.557253 (+ 274us) client_negotiation.cc:240] Negotiation successful
0926 03:59:59.557293 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9778,"thread_start_us":38,"threads_started":1}
I20260926 03:59:59.557387 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d57344652d534e21ab070ac069a86015" candidate_uuid: "a666f62187f9490baca7bcd1bbd7a00b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93a00a04d3e54c79aeb555bfd2ada656"
I20260926 03:59:59.557462 12040 raft_consensus.cc:3037] T d57344652d534e21ab070ac069a86015 P 93a00a04d3e54c79aeb555bfd2ada656 [term 0 FOLLOWER]: Advancing to term 1
I20260926 03:59:59.558413 12040 raft_consensus.cc:2454] T d57344652d534e21ab070ac069a86015 P 93a00a04d3e54c79aeb555bfd2ada656 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a666f62187f9490baca7bcd1bbd7a00b in term 1.
I20260926 03:59:59.558684 11838 leader_election.cc:302] T d57344652d534e21ab070ac069a86015 P a666f62187f9490baca7bcd1bbd7a00b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93a00a04d3e54c79aeb555bfd2ada656, a666f62187f9490baca7bcd1bbd7a00b; no voters:
I20260926 03:59:59.558835 11768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d57344652d534e21ab070ac069a86015" candidate_uuid: "a666f62187f9490baca7bcd1bbd7a00b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5434739be5a4d488b286f8c0307eff9" is_pre_election: true
I20260926 03:59:59.558951 11768 raft_consensus.cc:2379] T d57344652d534e21ab070ac069a86015 P e5434739be5a4d488b286f8c0307eff9 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a666f62187f9490baca7bcd1bbd7a00b in current term 1: Already voted for candidate e5434739be5a4d488b286f8c0307eff9 in this term.
I20260926 03:59:59.559130 11767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d57344652d534e21ab070ac069a86015" candidate_uuid: "a666f62187f9490baca7bcd1bbd7a00b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5434739be5a4d488b286f8c0307eff9"
I20260926 03:59:59.559206 11767 raft_consensus.cc:2379] T d57344652d534e21ab070ac069a86015 P e5434739be5a4d488b286f8c0307eff9 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate a666f62187f9490baca7bcd1bbd7a00b in current term 1: Already voted for candidate e5434739be5a4d488b286f8c0307eff9 in this term.
I20260926 03:59:59.559666 12112 raft_consensus.cc:2781] T d57344652d534e21ab070ac069a86015 P a666f62187f9490baca7bcd1bbd7a00b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 03:59:59.559729 11906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d57344652d534e21ab070ac069a86015" candidate_uuid: "e5434739be5a4d488b286f8c0307eff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a666f62187f9490baca7bcd1bbd7a00b"
I20260926 03:59:59.559828 12112 raft_consensus.cc:686] T d57344652d534e21ab070ac069a86015 P a666f62187f9490baca7bcd1bbd7a00b [term 1 LEADER]: Becoming Leader. State: Replica: a666f62187f9490baca7bcd1bbd7a00b, State: Running, Role: LEADER
I20260926 03:59:59.559919 12112 consensus_queue.cc:237] T d57344652d534e21ab070ac069a86015 P a666f62187f9490baca7bcd1bbd7a00b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5434739be5a4d488b286f8c0307eff9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39887 } } peers { permanent_uuid: "a666f62187f9490baca7bcd1bbd7a00b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38559 } } peers { permanent_uuid: "93a00a04d3e54c79aeb555bfd2ada656" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45905 } }
I20260926 03:59:59.560829 11601 catalog_manager.cc:5495] T d57344652d534e21ab070ac069a86015 P a666f62187f9490baca7bcd1bbd7a00b reported cstate change: term changed from 0 to 1, leader changed from <none> to a666f62187f9490baca7bcd1bbd7a00b (127.0.98.2). New cstate: current_term: 1 leader_uuid: "a666f62187f9490baca7bcd1bbd7a00b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5434739be5a4d488b286f8c0307eff9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a666f62187f9490baca7bcd1bbd7a00b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38559 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "93a00a04d3e54c79aeb555bfd2ada656" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45905 } health_report { overall_health: UNKNOWN } } }
I20260926 03:59:59.564908 12102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:59.544884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48421 (local address 127.0.98.3:45905)
0926 03:59:59.554617 (+ 9733us) server_negotiation.cc:206] Beginning negotiation
0926 03:59:59.554619 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:59:59.554625 (+ 6us) server_negotiation.cc:407] Connection header received
0926 03:59:59.554644 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:59:59.554644 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:59:59.554653 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:59:59.554702 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 03:59:59.562400 (+ 7698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:59.563042 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:59.563923 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:59.564462 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:59.564538 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:59.564548 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 03:59:59.564800 (+ 252us) server_negotiation.cc:299] Negotiation successful
0926 03:59:59.564839 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9720,"thread_start_us":67,"threads_started":1}
I20260926 03:59:59.564919 12118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:59.544709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45905 (local address 127.0.98.1:48421)
0926 03:59:59.545422 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0926 03:59:59.545433 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 03:59:59.545496 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 03:59:59.562241 (+ 16745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 03:59:59.562242 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 03:59:59.562244 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 03:59:59.562305 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:59.562307 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:59.563128 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:59.563129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:59.563828 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 03:59:59.563829 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 03:59:59.564565 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 03:59:59.564566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 03:59:59.564660 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:59.564667 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 03:59:59.564783 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 03:59:59.564827 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":691,"thread_start_us":90,"threads_started":1}
I20260926 03:59:59.565151 12039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d57344652d534e21ab070ac069a86015" candidate_uuid: "e5434739be5a4d488b286f8c0307eff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93a00a04d3e54c79aeb555bfd2ada656"
I20260926 03:59:59.565270 12039 raft_consensus.cc:2379] T d57344652d534e21ab070ac069a86015 P 93a00a04d3e54c79aeb555bfd2ada656 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate e5434739be5a4d488b286f8c0307eff9 in current term 1: Already voted for candidate a666f62187f9490baca7bcd1bbd7a00b in this term.
I20260926 03:59:59.565485 11701 leader_election.cc:302] T d57344652d534e21ab070ac069a86015 P e5434739be5a4d488b286f8c0307eff9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e5434739be5a4d488b286f8c0307eff9; no voters: 93a00a04d3e54c79aeb555bfd2ada656, a666f62187f9490baca7bcd1bbd7a00b
I20260926 03:59:59.565173 12040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d57344652d534e21ab070ac069a86015" candidate_uuid: "e5434739be5a4d488b286f8c0307eff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93a00a04d3e54c79aeb555bfd2ada656" is_pre_election: true
I20260926 03:59:59.565557 12040 raft_consensus.cc:2379] T d57344652d534e21ab070ac069a86015 P 93a00a04d3e54c79aeb555bfd2ada656 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e5434739be5a4d488b286f8c0307eff9 in current term 1: Already voted for candidate a666f62187f9490baca7bcd1bbd7a00b in this term.
I20260926 03:59:59.565624 12111 raft_consensus.cc:2726] T d57344652d534e21ab070ac069a86015 P e5434739be5a4d488b286f8c0307eff9 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260926 03:59:59.588110 12104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:59.585112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60730 (local address 127.0.98.2:38559)
0926 03:59:59.585155 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 03:59:59.585157 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 03:59:59.585165 (+ 8us) server_negotiation.cc:407] Connection header received
0926 03:59:59.585273 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 03:59:59.585274 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 03:59:59.585296 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 03:59:59.585362 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 03:59:59.585550 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:59.586142 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:59.587005 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 03:59:59.587292 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 03:59:59.587357 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 03:59:59.587522 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 03:59:59.587657 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 03:59:59.587749 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 03:59:59.587984 (+ 235us) server_negotiation.cc:299] Negotiation successful
0926 03:59:59.588008 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 03:59:59.590543 12040 raft_consensus.cc:1261] T d57344652d534e21ab070ac069a86015 P 93a00a04d3e54c79aeb555bfd2ada656 [term 1 FOLLOWER]: Refusing update from remote peer a666f62187f9490baca7bcd1bbd7a00b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:59:59.590942 12120 consensus_queue.cc:1035] T d57344652d534e21ab070ac069a86015 P a666f62187f9490baca7bcd1bbd7a00b [LEADER]: Connected to new peer: Peer: permanent_uuid: "93a00a04d3e54c79aeb555bfd2ada656" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45905 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:59:59.591020 11767 raft_consensus.cc:1261] T d57344652d534e21ab070ac069a86015 P e5434739be5a4d488b286f8c0307eff9 [term 1 FOLLOWER]: Refusing update from remote peer a666f62187f9490baca7bcd1bbd7a00b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 03:59:59.592322 12120 consensus_queue.cc:1035] T d57344652d534e21ab070ac069a86015 P a666f62187f9490baca7bcd1bbd7a00b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5434739be5a4d488b286f8c0307eff9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 03:59:59.594316 12124 mvcc.cc:205] Tried to move back new op lower bound from 7333458737518145536 to 7333458737397899264. Current Snapshot: MvccSnapshot[applied={T|T < 7333458737518145536}]
I20260926 03:59:59.595978 12123 mvcc.cc:205] Tried to move back new op lower bound from 7333458737518145536 to 7333458737397899264. Current Snapshot: MvccSnapshot[applied={T|T < 7333458737518145536}]
I20260926 03:59:59.598240 12126 mvcc.cc:205] Tried to move back new op lower bound from 7333458737518145536 to 7333458737397899264. Current Snapshot: MvccSnapshot[applied={T|T < 7333458737518145536}]
I20260926 03:59:59.600939 11601 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49858:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:59:59.601042 11601 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49858:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 03:59:59.606436 11601 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4754d31874144ebf91ef26140b8674da: Sending DeleteTablet for 3 replicas of tablet d57344652d534e21ab070ac069a86015
I20260926 03:59:59.606835 11748 tablet_service.cc:1514] Processing DeleteTablet for tablet d57344652d534e21ab070ac069a86015 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:59:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59058
I20260926 03:59:59.606869 12020 tablet_service.cc:1514] Processing DeleteTablet for tablet d57344652d534e21ab070ac069a86015 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:59:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41424
I20260926 03:59:59.607291 11886 tablet_service.cc:1514] Processing DeleteTablet for tablet d57344652d534e21ab070ac069a86015 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 03:59:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60726
I20260926 03:59:59.607555 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 11683
I20260926 03:59:59.612133 12133 tablet_replica.cc:330] T d57344652d534e21ab070ac069a86015 P 93a00a04d3e54c79aeb555bfd2ada656: stopping tablet replica
I20260926 03:59:59.612306 12133 raft_consensus.cc:2229] T d57344652d534e21ab070ac069a86015 P 93a00a04d3e54c79aeb555bfd2ada656 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 03:59:59.612504 12133 raft_consensus.cc:2258] T d57344652d534e21ab070ac069a86015 P 93a00a04d3e54c79aeb555bfd2ada656 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:59:59.612872 12135 tablet_replica.cc:330] T d57344652d534e21ab070ac069a86015 P a666f62187f9490baca7bcd1bbd7a00b: stopping tablet replica
I20260926 03:59:59.613058 12135 raft_consensus.cc:2229] T d57344652d534e21ab070ac069a86015 P a666f62187f9490baca7bcd1bbd7a00b [term 1 LEADER]: Raft consensus shutting down.
I20260926 03:59:59.613212 12135 raft_consensus.cc:2258] T d57344652d534e21ab070ac069a86015 P a666f62187f9490baca7bcd1bbd7a00b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 03:59:59.614316 12135 ts_tablet_manager.cc:1905] T d57344652d534e21ab070ac069a86015 P a666f62187f9490baca7bcd1bbd7a00b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:59:59.615556 12133 ts_tablet_manager.cc:1905] T d57344652d534e21ab070ac069a86015 P 93a00a04d3e54c79aeb555bfd2ada656: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 03:59:59.617934 12135 ts_tablet_manager.cc:1918] T d57344652d534e21ab070ac069a86015 P a666f62187f9490baca7bcd1bbd7a00b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:59:59.618000 12135 log.cc:1196] T d57344652d534e21ab070ac069a86015 P a666f62187f9490baca7bcd1bbd7a00b: Deleting WAL directory at /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/ts-1/wal/wals/d57344652d534e21ab070ac069a86015
I20260926 03:59:59.618255 12135 ts_tablet_manager.cc:1939] T d57344652d534e21ab070ac069a86015 P a666f62187f9490baca7bcd1bbd7a00b: Deleting consensus metadata
I20260926 03:59:59.619494 12133 ts_tablet_manager.cc:1918] T d57344652d534e21ab070ac069a86015 P 93a00a04d3e54c79aeb555bfd2ada656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 03:59:59.619520 11587 catalog_manager.cc:4841] TS a666f62187f9490baca7bcd1bbd7a00b (127.0.98.2:38559): tablet d57344652d534e21ab070ac069a86015 (table test-table [id=1b32a4cda64d490c9b09ff74319957b1]) successfully deleted
I20260926 03:59:59.619586 12133 log.cc:1196] T d57344652d534e21ab070ac069a86015 P 93a00a04d3e54c79aeb555bfd2ada656: Deleting WAL directory at /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790395095474643-392-0/minicluster-data/ts-2/wal/wals/d57344652d534e21ab070ac069a86015
I20260926 03:59:59.619845 12133 ts_tablet_manager.cc:1939] T d57344652d534e21ab070ac069a86015 P 93a00a04d3e54c79aeb555bfd2ada656: Deleting consensus metadata
I20260926 03:59:59.620253 11586 catalog_manager.cc:4841] TS 93a00a04d3e54c79aeb555bfd2ada656 (127.0.98.3:45905): tablet d57344652d534e21ab070ac069a86015 (table test-table [id=1b32a4cda64d490c9b09ff74319957b1]) successfully deleted
W20260926 03:59:59.624524 11589 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39887 (error 108)
I20260926 03:59:59.624751 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 11820
I20260926 03:59:59.626317 12103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 03:59:59.625482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39887 (local address 127.0.0.1:59070)
0926 03:59:59.626233 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0926 03:59:59.626282 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39887: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":723}
W20260926 03:59:59.626468 11589 catalog_manager.cc:4584] TS e5434739be5a4d488b286f8c0307eff9 (127.0.98.1:39887): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d57344652d534e21ab070ac069a86015: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39887: connect: Connection refused (error 111)
I20260926 03:59:59.629491 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 11957
I20260926 03:59:59.634917 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 11570
I20260926 03:59:59.642243 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 03:59:59.986578 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 03:59:59.986698 392 mini_ranger.cc:66] Stopping Ranger...
I20260926 04:00:00.331267 392 mini_ranger.cc:68] Stopped Ranger
2026-09-26 04:00:00.331 UTC [11156] LOG: received smart shutdown request
2026-09-26 04:00:00.337 UTC [11156] LOG: background worker "logical replication launcher" (PID 11165) exited with exit code 1
2026-09-26 04:00:00.337 UTC [11160] LOG: shutting down
2026-09-26 04:00:00.612 UTC [11156] LOG: database system is shut down
2026-09-26T04:00:00Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46382 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 04:00:00 dist-test-slave-nhpw krb5kdc[12145](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 04:00:00 dist-test-slave-nhpw krb5kdc[12145](info): set up 2 sockets
Sep 26 04:00:00 dist-test-slave-nhpw krb5kdc[12145](info): commencing operation
krb5kdc: starting...
W20260926 04:00:01.784711 392 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/krb5kdc/test-user.keytab.
Sep 26 04:00:01 dist-test-slave-nhpw krb5kdc[12145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395201, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T04:00:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:00:01Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260926 04:00:01.951115 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task7pf3XC/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/postgres -l logfile start
2026-09-26 04:00:04.581 UTC [12175] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 04:00:04.581 UTC [12175] LOG: listening on IPv4 address "127.0.98.20", port 41071
2026-09-26 04:00:04.584 UTC [12175] LOG: listening on Unix socket "/tmp/.s.PGSQL.41071"
2026-09-26 04:00:04.614 UTC [12178] LOG: database system was shut down at 2026-09-26 04:00:03 UTC
2026-09-26 04:00:04.618 UTC [12175] LOG: database system is ready to accept connections
I20260926 04:00:05.564529 392 mini_postgres.cc:96] Postgres bound to 41071
2026-09-26 04:00:05.570 UTC [12186] FATAL: database "slave" does not exist
127.0.98.20:41071 - accepting connections
I20260926 04:00:05.571471 392 mini_ranger.cc:161] Starting Ranger...
I20260926 04:00:05.586014 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 04:00:06.116716 392 mini_ranger.cc:87] Created ranger Postgres database
I20260926 04:00:06.116811 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ranger-admin
2026-09-26 04:00:06,377 [I] DB FLAVOR :POSTGRES
2026-09-26 04:00:06,378 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 04:00:06,378 [I] Checking connection..
2026-09-26 04:00:06,378 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7pf3XC/build/release/bin/postgresql.jar:/tmp/dist-test-task7pf3XC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41071/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 04:00:06,627 [I] Checking connection passed.
2026-09-26 04:00:06,628 [I] --------- Verifying version history table ---------
2026-09-26 04:00:06,628 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7pf3XC/build/release/bin/postgresql.jar:/tmp/dist-test-task7pf3XC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41071/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 04:00:06,887 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 04:00:06,887 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7pf3XC/build/release/bin/postgresql.jar:/tmp/dist-test-task7pf3XC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41071/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 04:00:07,129 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 04:00:07,129 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 04:00:07,129 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7pf3XC/build/release/bin/postgresql.jar:/tmp/dist-test-task7pf3XC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41071/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task7pf3XC/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 04:00:07.369 UTC [12250] WARNING: there is no transaction in progress
2026-09-26 04:00:07,376 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 04:00:07,376 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7pf3XC/build/release/bin/postgresql.jar:/tmp/dist-test-task7pf3XC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41071/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 04:00:07,613 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 04:00:07,614 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 04:00:07,614 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7pf3XC/build/release/bin/postgresql.jar:/tmp/dist-test-task7pf3XC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41071/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 04:00:07,867 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7pf3XC/build/release/bin/postgresql.jar:/tmp/dist-test-task7pf3XC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41071/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 04:00:08,121 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7pf3XC/build/release/bin/postgresql.jar:/tmp/dist-test-task7pf3XC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41071/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-nhpw.c.gcp-upstream.internal','N') ;"
2026-09-26 04:00:08,366 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7pf3XC/build/release/bin/postgresql.jar:/tmp/dist-test-task7pf3XC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41071/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 04:00:08,623 [I] Table x_portal_user does not exist in database ranger
2026-09-26 04:00:08,624 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7pf3XC/build/release/bin/postgresql.jar:/tmp/dist-test-task7pf3XC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41071/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 04:00:08,877 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 04:00:08,877 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7pf3XC/build/release/bin/postgresql.jar:/tmp/dist-test-task7pf3XC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41071/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 04:00:09,140 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7pf3XC/build/release/bin/postgresql.jar:/tmp/dist-test-task7pf3XC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41071/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 04:00:09,382 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 04:00:09,382 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7pf3XC/build/release/bin/postgresql.jar:/tmp/dist-test-task7pf3XC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41071/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task7pf3XC/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 04:00:10.374 UTC [12358] WARNING: there is no transaction in progress
2026-09-26 04:00:10.391 UTC [12358] WARNING: there is no transaction in progress
2026-09-26 04:00:10.409 UTC [12358] WARNING: there is no transaction in progress
2026-09-26 04:00:10.425 UTC [12358] WARNING: there is no transaction in progress
2026-09-26 04:00:10.443 UTC [12358] WARNING: there is no transaction in progress
2026-09-26 04:00:10.464 UTC [12358] WARNING: there is no transaction in progress
2026-09-26 04:00:10.521 UTC [12358] WARNING: there is no transaction in progress
2026-09-26 04:00:10.528 UTC [12358] WARNING: there is no transaction in progress
2026-09-26 04:00:10.543 UTC [12358] WARNING: there is no transaction in progress
2026-09-26 04:00:10.553 UTC [12358] WARNING: there is no transaction in progress
2026-09-26 04:00:10.564 UTC [12358] WARNING: there is no transaction in progress
2026-09-26 04:00:10.578 UTC [12358] WARNING: there is no transaction in progress
2026-09-26 04:00:10.588 UTC [12358] WARNING: there is no transaction in progress
2026-09-26 04:00:10.594 UTC [12358] WARNING: there is no transaction in progress
2026-09-26 04:00:10.601 UTC [12358] WARNING: there is no transaction in progress
2026-09-26 04:00:10.608 UTC [12358] WARNING: there is no transaction in progress
2026-09-26 04:00:11,288 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 04:00:11,288 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7pf3XC/build/release/bin/postgresql.jar:/tmp/dist-test-task7pf3XC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41071/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 04:00:11,512 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 04:00:11,512 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7pf3XC/build/release/bin/postgresql.jar:/tmp/dist-test-task7pf3XC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41071/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 04:00:11,774 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 04:00:11,775 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7pf3XC/build/release/bin/postgresql.jar:/tmp/dist-test-task7pf3XC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41071/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 04:00:12,024 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 04:00:12,024 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7pf3XC/build/release/bin/postgresql.jar:/tmp/dist-test-task7pf3XC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41071/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 04:00:12,269 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7pf3XC/build/release/bin/postgresql.jar:/tmp/dist-test-task7pf3XC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41071/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 04:00:12,514 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7pf3XC/build/release/bin/postgresql.jar:/tmp/dist-test-task7pf3XC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41071/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-nhpw.c.gcp-upstream.internal';"
2026-09-26 04:00:12,745 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 04:00:12,746 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7pf3XC/build/release/bin/postgresql.jar:/tmp/dist-test-task7pf3XC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41071/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 04:00:13,006 [I] DB_PATCHES have already been applied
I20260926 04:00:13.011809 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ranger-admin:/tmp/dist-test-task7pf3XC/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task7pf3XC/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task7pf3XC/build/release/bin/postgresql.jar:/tmp/dist-test-task7pf3XC/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 04:00:13.011919 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260926 04:00:13.015098 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:53085
Sep 26, 2026 4:00:13 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 4:00:13 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 4:00:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 4:00:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 4:00:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 4:00:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 4:00:13 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 04:00:14.133701 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.562s user 0.000s sys 0.009s
Sep 26, 2026 4:00:34 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 4:00:34 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 26, 2026 4:00:34 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 4:00:34 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 4:00:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 4:00:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 4:00:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 4:00:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 4:00:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 4:00:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 4:00:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 4:00:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 4:00:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 4:00:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 4:00:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 4:00:34 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 4:00:34 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 4:00:35 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 04:00:35.920563 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 04:00:36 dist-test-slave-nhpw krb5kdc[12145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395236, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 04:00:36.010663 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 04:00:36.021564 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 04:00:36.999073 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 04:00:36.999143 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ranger-kms
I20260926 04:00:36.999156 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:41071
2026-09-26 04:00:37,161 [I] DB FLAVOR :POSTGRES
2026-09-26 04:00:37,161 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 04:00:37,161 [I] Checking connection
2026-09-26 04:00:37,161 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7pf3XC/build/release/bin/postgresql.jar:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41071/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 04:00:37,395 [I] connection success
2026-09-26 04:00:37,395 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 04:00:37,395 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 04:00:37,395 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7pf3XC/build/release/bin/postgresql.jar:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41071/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 04:00:37,626 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 04:00:37,626 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 04:00:37,626 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 04:00:37,626 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task7pf3XC/build/release/bin/postgresql.jar:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41071/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 04:00:37,885 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 04:00:38.333587 392 mini_ranger_kms.cc:310] Created kms service
I20260926 04:00:38.552920 392 mini_ranger_kms.cc:326] Created kudu user
I20260926 04:00:38.601769 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 04:00:38 dist-test-slave-nhpw krb5kdc[12145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395236, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 04:00:39.037021 392 mini_ranger_kms.cc:384] Added ranger policy
I20260926 04:00:39.037141 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ranger-kms:/tmp/dist-test-task7pf3XC/build/release/bin/postgresql.jar:/tmp/dist-test-task7pf3XC/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task7pf3XC/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 04:00:39.037161 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260926 04:00:39.040450 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12538
I20260926 04:00:39.040529 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:55793
Sep 26, 2026 4:00:39 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 4:00:39 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 4:00:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 4:00:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 4:00:40 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 4:00:40 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 26 04:00:41 dist-test-slave-nhpw krb5kdc[12145](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790395241, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:00:41.325089 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.314s user 0.008s sys 0.137s
I20260926 04:00:41.325203 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 04:00:41.325249 392 mini_ranger_kms.cc:401] 127.0.98.20:55793/kms/v1/keys
Sep 26 04:00:41 dist-test-slave-nhpw krb5kdc[12145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395241, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 26 04:00:41 dist-test-slave-nhpw krb5kdc[12145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395241, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 26 04:00:44 dist-test-slave-nhpw krb5kdc[12145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 04:00:44.235881 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44347
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44347
--ranger_config_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:55793/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:00:44.336558 12589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:00:44.336776 12589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:00:44.336798 12589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:00:44.337934 12589 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:00:44.337976 12589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:00:44.337991 12589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:00:44.338003 12589 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:00:44.338016 12589 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:00:44.339607 12589 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:55793/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44347
--ranger_config_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:00:44.339854 12589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:00:44.340090 12589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 04:00:44.345496 12589 server_base.cc:1034] running on GCE node
W20260926 04:00:44.345655 12594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:00:44.345655 12595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:00:44.345727 12597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:00:44.346024 12589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:00:44.346292 12589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:00:44.347474 12589 hybrid_clock.cc:648] HybridClock initialized: now 1790395244347451 us; error 37 us; skew 500 ppm
Sep 26 04:00:44 dist-test-slave-nhpw krb5kdc[12145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395244, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:00:44.353627 12589 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 04:00:44.354038 12589 webserver.cc:460] Webserver started at http://127.0.98.62:36757/ using document root <none> and password file <none>
I20260926 04:00:44.354269 12589 fs_manager.cc:359] Metadata directory not provided
I20260926 04:00:44.354327 12589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:00:44.354478 12589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 04:00:44 dist-test-slave-nhpw krb5kdc[12145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395244, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 04:00:44.411892 12589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/master-0/data/instance:
uuid: "078e9a0c301348fdaa2792803826dae9"
format_stamp: "Formatted at 2026-09-26 04:00:44 on dist-test-slave-nhpw"
server_key: "862f0d41a708738cf2a55598aa34c1f5"
server_key_iv: "c577231b58f885238513388f4c2f2c76"
server_key_version: "kuduclusterkey@0"
I20260926 04:00:44.412376 12589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/master-0/wal/instance:
uuid: "078e9a0c301348fdaa2792803826dae9"
format_stamp: "Formatted at 2026-09-26 04:00:44 on dist-test-slave-nhpw"
server_key: "862f0d41a708738cf2a55598aa34c1f5"
server_key_iv: "c577231b58f885238513388f4c2f2c76"
server_key_version: "kuduclusterkey@0"
I20260926 04:00:44.414392 12589 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260926 04:00:44.428942 12604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:00:44.429247 12589 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:00:44.429391 12589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/master-0/wal
uuid: "078e9a0c301348fdaa2792803826dae9"
format_stamp: "Formatted at 2026-09-26 04:00:44 on dist-test-slave-nhpw"
server_key: "862f0d41a708738cf2a55598aa34c1f5"
server_key_iv: "c577231b58f885238513388f4c2f2c76"
server_key_version: "kuduclusterkey@0"
I20260926 04:00:44.429498 12589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:00:44.462832 12589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:00:44.464620 12589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:00:44.464876 12589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:00:44.469249 12589 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44347
I20260926 04:00:44.469323 12656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44347 every 8 connection(s)
I20260926 04:00:44.469663 12589 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
I20260926 04:00:44.470358 12657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:00:44.472887 12657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 078e9a0c301348fdaa2792803826dae9: Bootstrap starting.
I20260926 04:00:44.472957 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 12589
I20260926 04:00:44.473114 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/master-0/wal/instance
I20260926 04:00:44.473286 392 external_mini_cluster.cc:1428] Setting key ac05276b8d2259a6d88f7fb2801eebdf
I20260926 04:00:44.473639 12657 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 078e9a0c301348fdaa2792803826dae9: Neither blocks nor log segments found. Creating new log.
I20260926 04:00:44.474018 12657 log.cc:824] T 00000000000000000000000000000000 P 078e9a0c301348fdaa2792803826dae9: Log is configured to *not* fsync() on all Append() calls
I20260926 04:00:44.474809 12657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 078e9a0c301348fdaa2792803826dae9: No bootstrap required, opened a new log
I20260926 04:00:44.476899 12657 raft_consensus.cc:348] T 00000000000000000000000000000000 P 078e9a0c301348fdaa2792803826dae9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "078e9a0c301348fdaa2792803826dae9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44347 } }
I20260926 04:00:44.477059 12657 raft_consensus.cc:374] T 00000000000000000000000000000000 P 078e9a0c301348fdaa2792803826dae9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:00:44.477087 12657 raft_consensus.cc:729] T 00000000000000000000000000000000 P 078e9a0c301348fdaa2792803826dae9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 078e9a0c301348fdaa2792803826dae9, State: Initialized, Role: FOLLOWER
I20260926 04:00:44.477231 12657 consensus_queue.cc:260] T 00000000000000000000000000000000 P 078e9a0c301348fdaa2792803826dae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "078e9a0c301348fdaa2792803826dae9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44347 } }
I20260926 04:00:44.477316 12657 raft_consensus.cc:388] T 00000000000000000000000000000000 P 078e9a0c301348fdaa2792803826dae9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:00:44.477375 12657 raft_consensus.cc:482] T 00000000000000000000000000000000 P 078e9a0c301348fdaa2792803826dae9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:00:44.477427 12657 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 078e9a0c301348fdaa2792803826dae9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:00:44.477994 12657 raft_consensus.cc:504] T 00000000000000000000000000000000 P 078e9a0c301348fdaa2792803826dae9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "078e9a0c301348fdaa2792803826dae9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44347 } }
I20260926 04:00:44.478121 12657 leader_election.cc:302] T 00000000000000000000000000000000 P 078e9a0c301348fdaa2792803826dae9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 078e9a0c301348fdaa2792803826dae9; no voters:
I20260926 04:00:44.478312 12657 leader_election.cc:288] T 00000000000000000000000000000000 P 078e9a0c301348fdaa2792803826dae9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 04:00:44.478345 12662 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 078e9a0c301348fdaa2792803826dae9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:00:44.478466 12662 raft_consensus.cc:686] T 00000000000000000000000000000000 P 078e9a0c301348fdaa2792803826dae9 [term 1 LEADER]: Becoming Leader. State: Replica: 078e9a0c301348fdaa2792803826dae9, State: Running, Role: LEADER
I20260926 04:00:44.478626 12657 sys_catalog.cc:564] T 00000000000000000000000000000000 P 078e9a0c301348fdaa2792803826dae9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:00:44.478614 12662 consensus_queue.cc:237] T 00000000000000000000000000000000 P 078e9a0c301348fdaa2792803826dae9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "078e9a0c301348fdaa2792803826dae9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44347 } }
I20260926 04:00:44.479153 12663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 078e9a0c301348fdaa2792803826dae9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 078e9a0c301348fdaa2792803826dae9. Latest consensus state: current_term: 1 leader_uuid: "078e9a0c301348fdaa2792803826dae9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "078e9a0c301348fdaa2792803826dae9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44347 } } }
I20260926 04:00:44.479332 12663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 078e9a0c301348fdaa2792803826dae9 [sys.catalog]: This master's current role is: LEADER
I20260926 04:00:44.479825 12671 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 26 04:00:44 dist-test-slave-nhpw krb5kdc[12145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 04:00:44.480441 12671 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:00:44.480567 12664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 078e9a0c301348fdaa2792803826dae9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "078e9a0c301348fdaa2792803826dae9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "078e9a0c301348fdaa2792803826dae9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44347 } } }
I20260926 04:00:44.480731 12664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 078e9a0c301348fdaa2792803826dae9 [sys.catalog]: This master's current role is: LEADER
I20260926 04:00:44.480762 12657 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 04:00:44.485433 12671 catalog_manager.cc:1347] Generated new cluster ID: 9cc8cf092d8e4aa4aa0fd089ec343e49
I20260926 04:00:44.485508 12671 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 04:00:44.488111 12660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:44.473811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48990 (local address 127.0.98.62:44347)
0926 04:00:44.474022 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 04:00:44.474032 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 04:00:44.474068 (+ 36us) server_negotiation.cc:407] Connection header received
0926 04:00:44.474644 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:00:44.474652 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:00:44.474703 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:00:44.474863 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:00:44.475589 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:44.476469 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:44.477475 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:44.477929 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:44.478069 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:44.481016 (+ 2947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:00:44.481021 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:00:44.481022 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:00:44.481036 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:00:44.486041 (+ 5005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:00:44.486476 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:00:44.486479 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:00:44.486480 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:00:44.486589 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:00:44.486795 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:00:44.486795 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:00:44.486796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:00:44.487095 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:00:44.487658 (+ 563us) server_negotiation.cc:1035] Waiting for connection context
0926 04:00:44.487737 (+ 79us) server_negotiation.cc:299] Negotiation successful
0926 04:00:44.487884 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":106,"threads_started":1}
I20260926 04:00:44.503044 12671 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:00:44.503829 12671 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:00:44.512691 12671 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 078e9a0c301348fdaa2792803826dae9: Generated new TSK 0
I20260926 04:00:44.512912 12671 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 04:00:45.221835 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:55793/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44347
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:00:45.327652 12702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:00:45.327874 12702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:00:45.327896 12702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:00:45.329142 12702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:00:45.329182 12702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:00:45.329209 12702 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 04:00:45.330638 12702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:55793/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44347
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:00:45.330864 12702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:00:45.331116 12702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:00:45.337097 12710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:00:45.337070 12708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:00:45.337069 12707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:00:45.337493 12702 server_base.cc:1034] running on GCE node
I20260926 04:00:45.337697 12702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:00:45.337935 12702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:00:45.339112 12702 hybrid_clock.cc:648] HybridClock initialized: now 1790395245339098 us; error 35 us; skew 500 ppm
Sep 26 04:00:45 dist-test-slave-nhpw krb5kdc[12145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395245, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:00:45.341636 12702 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 04:00:45.342134 12702 webserver.cc:460] Webserver started at http://127.0.98.1:40989/ using document root <none> and password file <none>
I20260926 04:00:45.342365 12702 fs_manager.cc:359] Metadata directory not provided
I20260926 04:00:45.342408 12702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:00:45.342574 12702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 04:00:45 dist-test-slave-nhpw krb5kdc[12145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395245, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 04:00:45.356979 12702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-0/data/instance:
uuid: "9a11c2f492b1455a9369a077ac1536aa"
format_stamp: "Formatted at 2026-09-26 04:00:45 on dist-test-slave-nhpw"
server_key: "ee3669ed3ff3d2d8731f02d19164267e"
server_key_iv: "007178c14d0cc54f281982adb4314798"
server_key_version: "kuduclusterkey@0"
I20260926 04:00:45.358196 12702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9a11c2f492b1455a9369a077ac1536aa"
format_stamp: "Formatted at 2026-09-26 04:00:45 on dist-test-slave-nhpw"
server_key: "ee3669ed3ff3d2d8731f02d19164267e"
server_key_iv: "007178c14d0cc54f281982adb4314798"
server_key_version: "kuduclusterkey@0"
I20260926 04:00:45.360141 12702 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260926 04:00:45.376754 12717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:00:45.376995 12702 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 04:00:45.377103 12702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-0/wal
uuid: "9a11c2f492b1455a9369a077ac1536aa"
format_stamp: "Formatted at 2026-09-26 04:00:45 on dist-test-slave-nhpw"
server_key: "ee3669ed3ff3d2d8731f02d19164267e"
server_key_iv: "007178c14d0cc54f281982adb4314798"
server_key_version: "kuduclusterkey@0"
I20260926 04:00:45.377188 12702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:00:45.401818 12702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:00:45.404472 12702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:00:45.404664 12702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:00:45.405032 12702 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:00:45.405532 12702 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:00:45.405635 12702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:00:45.405740 12702 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:00:45.405774 12702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:00:45.412945 12702 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:32821
I20260926 04:00:45.413028 12832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:32821 every 8 connection(s)
I20260926 04:00:45.413376 12702 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
I20260926 04:00:45.419926 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 12702
I20260926 04:00:45.420030 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-0/wal/instance
I20260926 04:00:45.420226 392 external_mini_cluster.cc:1428] Setting key c41c43c715d9f8f2593528fbbb4e0c54
Sep 26 04:00:45 dist-test-slave-nhpw krb5kdc[12145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395245, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 04:00:45.427711 12835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:45.417711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42967 (local address 127.0.98.62:44347)
0926 04:00:45.417863 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 04:00:45.417865 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:00:45.418036 (+ 171us) server_negotiation.cc:407] Connection header received
0926 04:00:45.418584 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:00:45.418585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:00:45.418601 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:00:45.418679 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:00:45.419276 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:45.419792 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:45.421185 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:45.421400 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:45.421484 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:45.424149 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:00:45.424151 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:00:45.424151 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:00:45.424159 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:00:45.426087 (+ 1928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:00:45.426594 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:00:45.426595 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:00:45.426596 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:00:45.426672 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:00:45.426944 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:00:45.426945 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:00:45.426946 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:00:45.427080 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:00:45.427123 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 04:00:45.427499 (+ 376us) server_negotiation.cc:299] Negotiation successful
0926 04:00:45.427588 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":93,"threads_started":1}
I20260926 04:00:45.428447 12836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:45.417680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44347 (local address 127.0.98.1:42967)
0926 04:00:45.417895 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0926 04:00:45.417952 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 04:00:45.418467 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:00:45.418751 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:00:45.418755 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:00:45.418962 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:00:45.419174 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:00:45.419179 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:00:45.419888 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:00:45.419890 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:00:45.421082 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:00:45.421085 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:00:45.421510 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:00:45.421511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:00:45.421651 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:45.422232 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:00:45.422240 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:00:45.424029 (+ 1789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:00:45.426184 (+ 2155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:00:45.426191 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:00:45.426193 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:00:45.426500 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:00:45.426746 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:00:45.426747 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:00:45.426747 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:00:45.426870 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:00:45.427171 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:00:45.427176 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:00:45.427346 (+ 170us) client_negotiation.cc:769] Sending connection context
0926 04:00:45.428049 (+ 703us) client_negotiation.cc:240] Negotiation successful
0926 04:00:45.428213 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":101,"threads_started":1}
I20260926 04:00:45.429214 12833 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44347
I20260926 04:00:45.429304 12833 heartbeater.cc:464] Registering TS with master...
I20260926 04:00:45.429479 12833 heartbeater.cc:511] Master 127.0.98.62:44347 requested a full tablet report, sending...
I20260926 04:00:45.430071 12621 ts_manager.cc:194] Registered new tserver with Master: 9a11c2f492b1455a9369a077ac1536aa (127.0.98.1:32821)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 04:00:45.430979 12621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:42967
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 04:00:45.452021 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:55793/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44347
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:00:45.567317 12839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:00:45.567585 12839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:00:45.567618 12839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:00:45.569298 12839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:00:45.569356 12839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:00:45.569393 12839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 04:00:45.571491 12839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:55793/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44347
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:00:45.571734 12839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:00:45.572033 12839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:00:45.578452 12846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:00:45.578483 12847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:00:45.578977 12849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:00:45.579643 12839 server_base.cc:1034] running on GCE node
I20260926 04:00:45.579986 12839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:00:45.580242 12839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:00:45.581558 12839 hybrid_clock.cc:648] HybridClock initialized: now 1790395245581540 us; error 33 us; skew 500 ppm
Sep 26 04:00:45 dist-test-slave-nhpw krb5kdc[12145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395245, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:00:45.584839 12839 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 04:00:45.585384 12839 webserver.cc:460] Webserver started at http://127.0.98.2:44301/ using document root <none> and password file <none>
I20260926 04:00:45.585682 12839 fs_manager.cc:359] Metadata directory not provided
I20260926 04:00:45.585745 12839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:00:45.585922 12839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 04:00:45 dist-test-slave-nhpw krb5kdc[12145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395245, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 26 04:00:45 dist-test-slave-nhpw krb5kdc[12145](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790395245, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:00:45.601765 12839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-1/data/instance:
uuid: "cd33bc3d9b054802a4469efc639afa35"
format_stamp: "Formatted at 2026-09-26 04:00:45 on dist-test-slave-nhpw"
server_key: "f525a266b47c2f012cae924eb5ba1b13"
server_key_iv: "e7f418583566e210594c2fb0a2191052"
server_key_version: "kuduclusterkey@0"
I20260926 04:00:45.602192 12839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-1/wal/instance:
uuid: "cd33bc3d9b054802a4469efc639afa35"
format_stamp: "Formatted at 2026-09-26 04:00:45 on dist-test-slave-nhpw"
server_key: "f525a266b47c2f012cae924eb5ba1b13"
server_key_iv: "e7f418583566e210594c2fb0a2191052"
server_key_version: "kuduclusterkey@0"
I20260926 04:00:45.604784 12839 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260926 04:00:45.620478 12856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:00:45.620767 12839 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 04:00:45.620860 12839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-1/wal
uuid: "cd33bc3d9b054802a4469efc639afa35"
format_stamp: "Formatted at 2026-09-26 04:00:45 on dist-test-slave-nhpw"
server_key: "f525a266b47c2f012cae924eb5ba1b13"
server_key_iv: "e7f418583566e210594c2fb0a2191052"
server_key_version: "kuduclusterkey@0"
I20260926 04:00:45.620952 12839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:00:45.630991 12839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:00:45.633740 12839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:00:45.633939 12839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:00:45.634171 12839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:00:45.634563 12839 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:00:45.634651 12839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:00:45.634722 12839 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:00:45.634778 12839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:00:45.641106 12839 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46447
I20260926 04:00:45.641573 12839 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
I20260926 04:00:45.644948 12969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46447 every 8 connection(s)
I20260926 04:00:45.648908 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 12839
I20260926 04:00:45.649000 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-1/wal/instance
I20260926 04:00:45.649199 392 external_mini_cluster.cc:1428] Setting key df0f884c9e56052b0684b8649f903139
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Sep 26 04:00:45 dist-test-slave-nhpw krb5kdc[12145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395245, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 04:00:45.668075 12972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:45.645623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44347 (local address 127.0.98.2:44233)
0926 04:00:45.656759 (+ 11136us) negotiation.cc:107] Waiting for socket to connect
0926 04:00:45.656831 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0926 04:00:45.657642 (+ 811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:00:45.658049 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:00:45.658054 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:00:45.658309 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:00:45.658621 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:00:45.658627 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:00:45.659702 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:00:45.659704 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:00:45.661049 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:00:45.661052 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:00:45.661681 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:00:45.661681 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:00:45.661873 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:45.662497 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:00:45.662508 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:00:45.664262 (+ 1754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:00:45.666297 (+ 2035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:00:45.666306 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:00:45.666309 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:00:45.666619 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:00:45.666856 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:00:45.666857 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:00:45.666858 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:00:45.666994 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:00:45.667334 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:00:45.667341 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:00:45.667553 (+ 212us) client_negotiation.cc:769] Sending connection context
0926 04:00:45.667668 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 04:00:45.667836 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11072,"thread_start_us":89,"threads_started":1}
I20260926 04:00:45.668887 12835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:45.645905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44233 (local address 127.0.98.62:44347)
0926 04:00:45.645958 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 04:00:45.645960 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:00:45.656943 (+ 10983us) server_negotiation.cc:407] Connection header received
0926 04:00:45.657790 (+ 847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:00:45.657791 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:00:45.657807 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:00:45.657842 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:00:45.658744 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:45.659574 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:45.661155 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:45.661488 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:45.661535 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:45.664386 (+ 2851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:00:45.664388 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:00:45.664388 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:00:45.664397 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:00:45.666191 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:00:45.666717 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:00:45.666719 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:00:45.666719 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:00:45.666783 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:00:45.667069 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:00:45.667070 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:00:45.667070 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:00:45.667229 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:00:45.667305 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 04:00:45.668696 (+ 1391us) server_negotiation.cc:299] Negotiation successful
0926 04:00:45.668796 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 04:00:45.669454 12970 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44347
I20260926 04:00:45.669564 12970 heartbeater.cc:464] Registering TS with master...
I20260926 04:00:45.669756 12970 heartbeater.cc:511] Master 127.0.98.62:44347 requested a full tablet report, sending...
I20260926 04:00:45.670101 12621 ts_manager.cc:194] Registered new tserver with Master: cd33bc3d9b054802a4469efc639afa35 (127.0.98.2:46447)
I20260926 04:00:45.670591 12621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44233
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 04:00:45.682329 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:55793/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44347
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:00:45.802917 12975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:00:45.803253 12975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:00:45.803329 12975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:00:45.805071 12975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:00:45.805177 12975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:00:45.805353 12975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 04:00:45.807483 12975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:55793/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44347
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:00:45.807937 12975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:00:45.808230 12975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:00:45.814687 12981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:00:45.815017 12984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:00:45.815832 12975 server_base.cc:1034] running on GCE node
W20260926 04:00:45.817066 12982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:00:45.817337 12975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:00:45.817595 12975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:00:45.818760 12975 hybrid_clock.cc:648] HybridClock initialized: now 1790395245818740 us; error 33 us; skew 500 ppm
Sep 26 04:00:45 dist-test-slave-nhpw krb5kdc[12145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395245, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:00:45.822077 12975 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 04:00:45.822583 12975 webserver.cc:460] Webserver started at http://127.0.98.3:42403/ using document root <none> and password file <none>
I20260926 04:00:45.822842 12975 fs_manager.cc:359] Metadata directory not provided
I20260926 04:00:45.822890 12975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:00:45.823012 12975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 04:00:45 dist-test-slave-nhpw krb5kdc[12145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395245, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 04:00:45.846416 12975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-2/data/instance:
uuid: "ff5b3d52cc2a4d93904265fedf382e36"
format_stamp: "Formatted at 2026-09-26 04:00:45 on dist-test-slave-nhpw"
server_key: "0a0710327b92b0f6dee0e4d89295b7e9"
server_key_iv: "140b32cbd070309965589fad84a06cd9"
server_key_version: "kuduclusterkey@0"
I20260926 04:00:45.846999 12975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ff5b3d52cc2a4d93904265fedf382e36"
format_stamp: "Formatted at 2026-09-26 04:00:45 on dist-test-slave-nhpw"
server_key: "0a0710327b92b0f6dee0e4d89295b7e9"
server_key_iv: "140b32cbd070309965589fad84a06cd9"
server_key_version: "kuduclusterkey@0"
I20260926 04:00:45.849771 12975 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260926 04:00:45.877894 12991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:00:45.878156 12975 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.000s
I20260926 04:00:45.878244 12975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-2/wal
uuid: "ff5b3d52cc2a4d93904265fedf382e36"
format_stamp: "Formatted at 2026-09-26 04:00:45 on dist-test-slave-nhpw"
server_key: "0a0710327b92b0f6dee0e4d89295b7e9"
server_key_iv: "140b32cbd070309965589fad84a06cd9"
server_key_version: "kuduclusterkey@0"
I20260926 04:00:45.878355 12975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:00:45.913303 12975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:00:45.915902 12975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:00:45.916090 12975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:00:45.916324 12975 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:00:45.916738 12975 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:00:45.916831 12975 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:00:45.916899 12975 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:00:45.916955 12975 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:00:45.923220 12975 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35537
I20260926 04:00:45.923660 12975 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
I20260926 04:00:45.924409 13104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35537 every 8 connection(s)
Sep 26 04:00:45 dist-test-slave-nhpw krb5kdc[12145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395245, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 04:00:45.932435 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 12975
I20260926 04:00:45.932533 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-2/wal/instance
I20260926 04:00:45.932744 392 external_mini_cluster.cc:1428] Setting key 202d3a1851b89adcf4cacef2b8bf9dc3
I20260926 04:00:45.935390 12835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:45.925092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39007 (local address 127.0.98.62:44347)
0926 04:00:45.925163 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 04:00:45.925165 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:00:45.925435 (+ 270us) server_negotiation.cc:407] Connection header received
0926 04:00:45.926130 (+ 695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:00:45.926132 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:00:45.926145 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:00:45.926215 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:00:45.927001 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:45.927761 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:45.929237 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:45.929515 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:45.929592 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:45.932064 (+ 2472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:00:45.932065 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:00:45.932066 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:00:45.932075 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:00:45.933856 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:00:45.934302 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:00:45.934303 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:00:45.934304 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:00:45.934364 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:00:45.934663 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:00:45.934664 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:00:45.934664 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:00:45.934802 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:00:45.934875 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 04:00:45.935211 (+ 336us) server_negotiation.cc:299] Negotiation successful
0926 04:00:45.935291 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 04:00:45.935735 13107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:45.925055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44347 (local address 127.0.98.3:39007)
0926 04:00:45.925242 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 04:00:45.925295 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 04:00:45.926000 (+ 705us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:00:45.926312 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:00:45.926316 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:00:45.926615 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:00:45.926895 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:00:45.926901 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:00:45.927870 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:00:45.927871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:00:45.929137 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:00:45.929140 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:00:45.929629 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:00:45.929630 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:00:45.929766 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:45.930349 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:00:45.930361 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:00:45.931953 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:00:45.933947 (+ 1994us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:00:45.933953 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:00:45.933956 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:00:45.934218 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:00:45.934443 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:00:45.934444 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:00:45.934444 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:00:45.934583 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:00:45.934880 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:00:45.934885 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:00:45.935052 (+ 167us) client_negotiation.cc:769] Sending connection context
0926 04:00:45.935167 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 04:00:45.935312 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":80,"threads_started":1}
I20260926 04:00:45.936147 13105 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44347
I20260926 04:00:45.936261 13105 heartbeater.cc:464] Registering TS with master...
I20260926 04:00:45.936443 13105 heartbeater.cc:511] Master 127.0.98.62:44347 requested a full tablet report, sending...
I20260926 04:00:45.936815 12621 ts_manager.cc:194] Registered new tserver with Master: ff5b3d52cc2a4d93904265fedf382e36 (127.0.98.3:35537)
I20260926 04:00:45.937312 12621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:39007
I20260926 04:00:45.944563 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 04:00:45.945300 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 12702
I20260926 04:00:45.951463 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 12839
I20260926 04:00:45.972397 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 12975
I20260926 04:00:45.979530 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 12589
I20260926 04:00:45.984359 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44347
--webserver_interface=127.0.98.62
--webserver_port=36757
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44347
--ranger_config_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:55793/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:55793/kms with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:00:46.081952 13108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:00:46.082258 13108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:00:46.082300 13108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:00:46.084416 13108 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:00:46.084472 13108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:00:46.084499 13108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:00:46.084583 13108 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:00:46.084618 13108 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:00:46.087568 13108 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:55793/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44347
--ranger_config_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44347
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=36757
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:00:46.087828 13108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:00:46.088155 13108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:00:46.093320 13114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:00:46.093328 13116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:00:46.094564 13113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:00:46.100672 13108 server_base.cc:1034] running on GCE node
I20260926 04:00:46.100940 13108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:00:46.101285 13108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:00:46.102581 13108 hybrid_clock.cc:648] HybridClock initialized: now 1790395246102563 us; error 32 us; skew 500 ppm
Sep 26 04:00:46 dist-test-slave-nhpw krb5kdc[12145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395246, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:00:46.105382 13108 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 04:00:46.105911 13108 webserver.cc:460] Webserver started at http://127.0.98.62:36757/ using document root <none> and password file <none>
I20260926 04:00:46.106148 13108 fs_manager.cc:359] Metadata directory not provided
I20260926 04:00:46.106215 13108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 04:00:46 dist-test-slave-nhpw krb5kdc[12145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395246, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 04:00:46.154084 13108 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:00:46.154819 13124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:00:46.155015 13108 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:00:46.155112 13108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/master-0/wal
uuid: "078e9a0c301348fdaa2792803826dae9"
format_stamp: "Formatted at 2026-09-26 04:00:44 on dist-test-slave-nhpw"
server_key: "862f0d41a708738cf2a55598aa34c1f5"
server_key_iv: "c577231b58f885238513388f4c2f2c76"
server_key_version: "kuduclusterkey@0"
I20260926 04:00:46.155471 13108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:00:46.193212 13108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:00:46.195140 13108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:00:46.195344 13108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:00:46.199587 13108 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44347
I20260926 04:00:46.199633 13176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44347 every 8 connection(s)
I20260926 04:00:46.199994 13108 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
I20260926 04:00:46.201186 13177 sys_catalog.cc:263] Verifying existing consensus state
I20260926 04:00:46.201839 13177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 078e9a0c301348fdaa2792803826dae9: Bootstrap starting.
I20260926 04:00:46.209494 13177 log.cc:824] T 00000000000000000000000000000000 P 078e9a0c301348fdaa2792803826dae9: Log is configured to *not* fsync() on all Append() calls
I20260926 04:00:46.210237 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 13108
I20260926 04:00:46.210523 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:55793/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:32821
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:44347
--webserver_port=40989
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:55793/kms with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 04:00:46.211604 13177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 078e9a0c301348fdaa2792803826dae9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 04:00:46.211913 13177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 078e9a0c301348fdaa2792803826dae9: Bootstrap complete.
I20260926 04:00:46.214394 13177 raft_consensus.cc:348] T 00000000000000000000000000000000 P 078e9a0c301348fdaa2792803826dae9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "078e9a0c301348fdaa2792803826dae9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44347 } }
I20260926 04:00:46.214644 13177 raft_consensus.cc:729] T 00000000000000000000000000000000 P 078e9a0c301348fdaa2792803826dae9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 078e9a0c301348fdaa2792803826dae9, State: Initialized, Role: FOLLOWER
I20260926 04:00:46.214846 13177 consensus_queue.cc:260] T 00000000000000000000000000000000 P 078e9a0c301348fdaa2792803826dae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "078e9a0c301348fdaa2792803826dae9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44347 } }
I20260926 04:00:46.214941 13177 raft_consensus.cc:388] T 00000000000000000000000000000000 P 078e9a0c301348fdaa2792803826dae9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:00:46.214993 13177 raft_consensus.cc:482] T 00000000000000000000000000000000 P 078e9a0c301348fdaa2792803826dae9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:00:46.215040 13177 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 078e9a0c301348fdaa2792803826dae9 [term 1 FOLLOWER]: Advancing to term 2
I20260926 04:00:46.216007 13177 raft_consensus.cc:504] T 00000000000000000000000000000000 P 078e9a0c301348fdaa2792803826dae9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "078e9a0c301348fdaa2792803826dae9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44347 } }
I20260926 04:00:46.216131 13177 leader_election.cc:302] T 00000000000000000000000000000000 P 078e9a0c301348fdaa2792803826dae9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 078e9a0c301348fdaa2792803826dae9; no voters:
I20260926 04:00:46.216302 13177 leader_election.cc:288] T 00000000000000000000000000000000 P 078e9a0c301348fdaa2792803826dae9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 04:00:46.216369 13181 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 078e9a0c301348fdaa2792803826dae9 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 04:00:46.216552 13177 sys_catalog.cc:564] T 00000000000000000000000000000000 P 078e9a0c301348fdaa2792803826dae9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:00:46.216594 13181 raft_consensus.cc:686] T 00000000000000000000000000000000 P 078e9a0c301348fdaa2792803826dae9 [term 2 LEADER]: Becoming Leader. State: Replica: 078e9a0c301348fdaa2792803826dae9, State: Running, Role: LEADER
I20260926 04:00:46.216812 13181 consensus_queue.cc:237] T 00000000000000000000000000000000 P 078e9a0c301348fdaa2792803826dae9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "078e9a0c301348fdaa2792803826dae9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44347 } }
I20260926 04:00:46.217154 13182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 078e9a0c301348fdaa2792803826dae9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "078e9a0c301348fdaa2792803826dae9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "078e9a0c301348fdaa2792803826dae9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44347 } } }
I20260926 04:00:46.217173 13183 sys_catalog.cc:455] T 00000000000000000000000000000000 P 078e9a0c301348fdaa2792803826dae9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 078e9a0c301348fdaa2792803826dae9. Latest consensus state: current_term: 2 leader_uuid: "078e9a0c301348fdaa2792803826dae9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "078e9a0c301348fdaa2792803826dae9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44347 } } }
I20260926 04:00:46.217237 13182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 078e9a0c301348fdaa2792803826dae9 [sys.catalog]: This master's current role is: LEADER
I20260926 04:00:46.217262 13183 sys_catalog.cc:458] T 00000000000000000000000000000000 P 078e9a0c301348fdaa2792803826dae9 [sys.catalog]: This master's current role is: LEADER
I20260926 04:00:46.217552 13190 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:00:46.218089 13190 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:00:46.218288 13190 catalog_manager.cc:1259] Loaded cluster ID: 9cc8cf092d8e4aa4aa0fd089ec343e49
I20260926 04:00:46.218369 13190 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 04:00:46.218397 13177 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 04:00:46.219842 13190 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:00:46.220067 13190 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 078e9a0c301348fdaa2792803826dae9: Loaded TSK: 0
I20260926 04:00:46.220319 13190 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 04:00:46.319411 13180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:00:46.319702 13180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:00:46.319794 13180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:00:46.321678 13180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:00:46.321748 13180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:00:46.321792 13180 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 04:00:46.324077 13180 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:55793/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:32821
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=40989
--tserver_master_addrs=127.0.98.62:44347
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13180
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:00:46.324342 13180 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:00:46.324672 13180 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:00:46.330888 13215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:00:46.330888 13214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:00:46.331162 13217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:00:46.331514 13180 server_base.cc:1034] running on GCE node
I20260926 04:00:46.331737 13180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:00:46.332011 13180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:00:46.333189 13180 hybrid_clock.cc:648] HybridClock initialized: now 1790395246333159 us; error 43 us; skew 500 ppm
Sep 26 04:00:46 dist-test-slave-nhpw krb5kdc[12145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395246, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:00:46.336263 13180 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 04:00:46.336817 13180 webserver.cc:460] Webserver started at http://127.0.98.1:40989/ using document root <none> and password file <none>
I20260926 04:00:46.337142 13180 fs_manager.cc:359] Metadata directory not provided
I20260926 04:00:46.337270 13180 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 04:00:46 dist-test-slave-nhpw krb5kdc[12145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395246, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 04:00:46.365183 13180 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 04:00:46.365928 13225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:00:46.366122 13180 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 04:00:46.366212 13180 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-0/wal
uuid: "9a11c2f492b1455a9369a077ac1536aa"
format_stamp: "Formatted at 2026-09-26 04:00:45 on dist-test-slave-nhpw"
server_key: "ee3669ed3ff3d2d8731f02d19164267e"
server_key_iv: "007178c14d0cc54f281982adb4314798"
server_key_version: "kuduclusterkey@0"
I20260926 04:00:46.366581 13180 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:00:46.374576 13180 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:00:46.381395 13180 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:00:46.381556 13180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:00:46.381814 13180 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:00:46.382161 13180 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:00:46.382216 13180 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:00:46.382282 13180 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:00:46.382332 13180 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:00:46.389389 13180 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:32821
I20260926 04:00:46.389883 13180 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
I20260926 04:00:46.390723 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 13180
I20260926 04:00:46.391000 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:55793/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:46447
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:44347
--webserver_port=44301
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:55793/kms with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 04:00:46.401391 13338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:32821 every 8 connection(s)
Sep 26 04:00:46 dist-test-slave-nhpw krb5kdc[12145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395246, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 04:00:46.416234 13341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:46.390656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44347 (local address 127.0.98.1:56871)
0926 04:00:46.391480 (+ 824us) negotiation.cc:107] Waiting for socket to connect
0926 04:00:46.391520 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 04:00:46.392114 (+ 594us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:00:46.405404 (+ 13290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:00:46.405409 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:00:46.405667 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:00:46.405942 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:00:46.405948 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:00:46.407666 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:00:46.407668 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:00:46.408961 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:00:46.408964 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:00:46.409735 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:00:46.409736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:00:46.409867 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:46.410494 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:00:46.410503 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:00:46.412136 (+ 1633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:00:46.414358 (+ 2222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:00:46.414364 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:00:46.414366 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:00:46.414641 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:00:46.414970 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:00:46.414971 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:00:46.414971 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:00:46.415086 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:00:46.415547 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:00:46.415553 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:00:46.415709 (+ 156us) client_negotiation.cc:769] Sending connection context
0926 04:00:46.415796 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 04:00:46.415950 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":783,"thread_start_us":543,"threads_started":1}
I20260926 04:00:46.417344 13342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:46.391506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56871 (local address 127.0.98.62:44347)
0926 04:00:46.404843 (+ 13337us) server_negotiation.cc:206] Beginning negotiation
0926 04:00:46.404854 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 04:00:46.404902 (+ 48us) server_negotiation.cc:407] Connection header received
0926 04:00:46.405070 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:00:46.405077 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:00:46.405158 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:00:46.406070 (+ 912us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:00:46.406178 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:46.407523 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:46.409072 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:46.409615 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:46.410165 (+ 550us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:46.412261 (+ 2096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:00:46.412266 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:00:46.412266 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:00:46.412279 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:00:46.414260 (+ 1981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:00:46.414734 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:00:46.414737 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:00:46.414737 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:00:46.414833 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:00:46.415174 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:00:46.415175 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:00:46.415175 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:00:46.415456 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:00:46.416910 (+ 1454us) server_negotiation.cc:1035] Waiting for connection context
0926 04:00:46.416986 (+ 76us) server_negotiation.cc:299] Negotiation successful
0926 04:00:46.417143 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13206,"thread_start_us":82,"threads_started":1}
I20260926 04:00:46.418275 13339 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44347
I20260926 04:00:46.418373 13339 heartbeater.cc:464] Registering TS with master...
I20260926 04:00:46.418561 13339 heartbeater.cc:511] Master 127.0.98.62:44347 requested a full tablet report, sending...
W20260926 04:00:46.419498 13339 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44347 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 04:00:46.419564 13339 heartbeater.cc:464] Registering TS with master...
I20260926 04:00:46.419627 13339 heartbeater.cc:511] Master 127.0.98.62:44347 requested a full tablet report, sending...
I20260926 04:00:46.420010 13339 heartbeater.cc:464] Registering TS with master...
I20260926 04:00:46.420082 13339 heartbeater.cc:511] Master 127.0.98.62:44347 requested a full tablet report, sending...
W20260926 04:00:46.420414 13339 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 04:00:46.515417 13343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:00:46.515682 13343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:00:46.515717 13343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:00:46.517553 13343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:00:46.517616 13343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:00:46.517663 13343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 04:00:46.519822 13343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:55793/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:46447
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=44301
--tserver_master_addrs=127.0.98.62:44347
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13343
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:00:46.520192 13343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:00:46.520473 13343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:00:46.526782 13349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:00:46.526886 13351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:00:46.526782 13348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:00:46.526969 13343 server_base.cc:1034] running on GCE node
I20260926 04:00:46.527343 13343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:00:46.527585 13343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:00:46.528718 13343 hybrid_clock.cc:648] HybridClock initialized: now 1790395246528701 us; error 32 us; skew 500 ppm
Sep 26 04:00:46 dist-test-slave-nhpw krb5kdc[12145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395246, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:00:46.531322 13343 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 04:00:46.532125 13343 webserver.cc:460] Webserver started at http://127.0.98.2:44301/ using document root <none> and password file <none>
I20260926 04:00:46.532388 13343 fs_manager.cc:359] Metadata directory not provided
I20260926 04:00:46.532473 13343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 04:00:46 dist-test-slave-nhpw krb5kdc[12145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395246, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 04:00:46.552428 13343 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260926 04:00:46.558866 13359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:00:46.559142 13343 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.000s sys 0.001s
I20260926 04:00:46.559248 13343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-1/wal
uuid: "cd33bc3d9b054802a4469efc639afa35"
format_stamp: "Formatted at 2026-09-26 04:00:45 on dist-test-slave-nhpw"
server_key: "f525a266b47c2f012cae924eb5ba1b13"
server_key_iv: "e7f418583566e210594c2fb0a2191052"
server_key_version: "kuduclusterkey@0"
I20260926 04:00:46.559751 13343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:00:46.571511 13343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:00:46.573998 13343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:00:46.574148 13343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:00:46.574390 13343 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:00:46.574746 13343 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:00:46.574795 13343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:00:46.574826 13343 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:00:46.574894 13343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:00:46.582022 13343 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46447
I20260926 04:00:46.582541 13343 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
I20260926 04:00:46.583050 13472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46447 every 8 connection(s)
I20260926 04:00:46.587244 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 13343
I20260926 04:00:46.587533 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:55793/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:35537
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:44347
--webserver_port=42403
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:55793/kms with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 04:00:46 dist-test-slave-nhpw krb5kdc[12145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395246, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 04:00:46.593771 13475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:46.583276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44347 (local address 127.0.98.2:54551)
0926 04:00:46.583423 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 04:00:46.583468 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 04:00:46.584102 (+ 634us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:00:46.584402 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:00:46.584406 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:00:46.584691 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:00:46.585121 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:00:46.585128 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:00:46.585858 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:00:46.585859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:00:46.587146 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:00:46.587148 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:00:46.587747 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:00:46.587748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:00:46.587888 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:46.588473 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:00:46.588483 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:00:46.589997 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:00:46.591946 (+ 1949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:00:46.591953 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:00:46.591955 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:00:46.592275 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:00:46.592492 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:00:46.592493 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:00:46.592494 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:00:46.592628 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:00:46.592978 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:00:46.592985 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:00:46.593189 (+ 204us) client_negotiation.cc:769] Sending connection context
0926 04:00:46.593302 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 04:00:46.593478 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":85,"threads_started":1}
I20260926 04:00:46.594310 13342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:46.583568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54551 (local address 127.0.98.62:44347)
0926 04:00:46.583612 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 04:00:46.583614 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:00:46.583626 (+ 12us) server_negotiation.cc:407] Connection header received
0926 04:00:46.584241 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:00:46.584243 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:00:46.584261 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:00:46.584333 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:00:46.585239 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:46.585749 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:46.587262 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:46.587590 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:46.587638 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:46.590104 (+ 2466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:00:46.590106 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:00:46.590106 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:00:46.590114 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:00:46.591851 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:00:46.592364 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:00:46.592365 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:00:46.592365 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:00:46.592427 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:00:46.592730 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:00:46.592731 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:00:46.592731 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:00:46.592883 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:00:46.592950 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 04:00:46.594054 (+ 1104us) server_negotiation.cc:299] Negotiation successful
0926 04:00:46.594175 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 04:00:46.594748 13473 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44347
I20260926 04:00:46.594872 13473 heartbeater.cc:464] Registering TS with master...
I20260926 04:00:46.595058 13473 heartbeater.cc:511] Master 127.0.98.62:44347 requested a full tablet report, sending...
W20260926 04:00:46.595778 13473 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44347 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 04:00:46.595861 13473 heartbeater.cc:464] Registering TS with master...
I20260926 04:00:46.595973 13473 heartbeater.cc:511] Master 127.0.98.62:44347 requested a full tablet report, sending...
I20260926 04:00:46.596308 13473 heartbeater.cc:464] Registering TS with master...
I20260926 04:00:46.596436 13473 heartbeater.cc:511] Master 127.0.98.62:44347 requested a full tablet report, sending...
W20260926 04:00:46.596903 13473 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 04:00:46.695075 13476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:00:46.695290 13476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:00:46.695325 13476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:00:46.696702 13476 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:00:46.696776 13476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:00:46.696820 13476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 04:00:46.698280 13476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:55793/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:35537
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=42403
--tserver_master_addrs=127.0.98.62:44347
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13476
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:00:46.698532 13476 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:00:46.698781 13476 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:00:46.703794 13481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:00:46.703794 13482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:00:46.703884 13484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:00:46.704530 13476 server_base.cc:1034] running on GCE node
I20260926 04:00:46.704772 13476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:00:46.705044 13476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:00:46.712422 13476 hybrid_clock.cc:648] HybridClock initialized: now 1790395246712396 us; error 35 us; skew 500 ppm
Sep 26 04:00:46 dist-test-slave-nhpw krb5kdc[12145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395246, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:00:46.715058 13476 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 04:00:46.715595 13476 webserver.cc:460] Webserver started at http://127.0.98.3:42403/ using document root <none> and password file <none>
I20260926 04:00:46.715864 13476 fs_manager.cc:359] Metadata directory not provided
I20260926 04:00:46.715943 13476 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 04:00:46 dist-test-slave-nhpw krb5kdc[12145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395246, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 04:00:46.739966 13476 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:00:46.740736 13493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:00:46.740938 13476 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 04:00:46.741025 13476 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-2/wal
uuid: "ff5b3d52cc2a4d93904265fedf382e36"
format_stamp: "Formatted at 2026-09-26 04:00:45 on dist-test-slave-nhpw"
server_key: "0a0710327b92b0f6dee0e4d89295b7e9"
server_key_iv: "140b32cbd070309965589fad84a06cd9"
server_key_version: "kuduclusterkey@0"
I20260926 04:00:46.741468 13476 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:00:46.759490 13476 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:00:46.761770 13476 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:00:46.761929 13476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:00:46.762171 13476 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:00:46.762475 13476 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:00:46.762528 13476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:00:46.762581 13476 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:00:46.762611 13476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:00:46.769426 13476 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35537
I20260926 04:00:46.769833 13476 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
I20260926 04:00:46.770438 13606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35537 every 8 connection(s)
Sep 26 04:00:46 dist-test-slave-nhpw krb5kdc[12145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395246, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 04:00:46.776690 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 13476
I20260926 04:00:46.780464 13342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:46.770955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46201 (local address 127.0.98.62:44347)
0926 04:00:46.771029 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 04:00:46.771031 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:00:46.771042 (+ 11us) server_negotiation.cc:407] Connection header received
0926 04:00:46.771447 (+ 405us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:00:46.771448 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:00:46.771459 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:00:46.771530 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:00:46.772031 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:46.772835 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:46.774265 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:46.774455 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:46.774516 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:46.776974 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:00:46.776975 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:00:46.776976 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:00:46.776986 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:00:46.778802 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:00:46.779345 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:00:46.779347 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:00:46.779348 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:00:46.779415 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:00:46.779699 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:00:46.779700 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:00:46.779700 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:00:46.779827 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:00:46.779885 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 04:00:46.780277 (+ 392us) server_negotiation.cc:299] Negotiation successful
0926 04:00:46.780367 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 04:00:46.780689 13609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:46.770634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44347 (local address 127.0.98.3:46201)
0926 04:00:46.770807 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 04:00:46.770839 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 04:00:46.771355 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:00:46.771589 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:00:46.771593 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:00:46.771768 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:00:46.771956 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:00:46.771959 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:00:46.772938 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:00:46.772939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:00:46.774173 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:00:46.774175 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:00:46.774549 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:00:46.774550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:00:46.774684 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:46.775160 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:00:46.775169 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:00:46.776801 (+ 1632us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:00:46.778903 (+ 2102us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:00:46.778910 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:00:46.778912 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:00:46.779247 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:00:46.779489 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:00:46.779490 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:00:46.779490 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:00:46.779623 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:00:46.779910 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:00:46.779915 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:00:46.780118 (+ 203us) client_negotiation.cc:769] Sending connection context
0926 04:00:46.780240 (+ 122us) client_negotiation.cc:240] Negotiation successful
0926 04:00:46.780393 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":90,"threads_started":1}
I20260926 04:00:46.784006 13607 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44347
I20260926 04:00:46.784112 13607 heartbeater.cc:464] Registering TS with master...
I20260926 04:00:46.784334 13607 heartbeater.cc:511] Master 127.0.98.62:44347 requested a full tablet report, sending...
W20260926 04:00:46.785107 13607 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:44347 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 04:00:46.785182 13607 heartbeater.cc:464] Registering TS with master...
I20260926 04:00:46.785252 13607 heartbeater.cc:511] Master 127.0.98.62:44347 requested a full tablet report, sending...
I20260926 04:00:46.785650 13607 heartbeater.cc:464] Registering TS with master...
I20260926 04:00:46.785730 13607 heartbeater.cc:511] Master 127.0.98.62:44347 requested a full tablet report, sending...
W20260926 04:00:46.786091 13607 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 04:00:46.788111 13611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:46.777671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49006 (local address 127.0.98.62:44347)
0926 04:00:46.777808 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 04:00:46.777811 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:00:46.777819 (+ 8us) server_negotiation.cc:407] Connection header received
0926 04:00:46.777841 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:00:46.777842 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:00:46.777855 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:00:46.777944 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:00:46.779023 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:46.779985 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:46.781672 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:46.782021 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:46.782107 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:46.783677 (+ 1570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:00:46.783679 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:00:46.783679 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:00:46.783687 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:00:46.786832 (+ 3145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:00:46.787234 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:00:46.787235 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:00:46.787235 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:00:46.787301 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:00:46.787523 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:00:46.787524 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:00:46.787524 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:00:46.787647 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:00:46.787827 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0926 04:00:46.787878 (+ 51us) server_negotiation.cc:299] Negotiation successful
0926 04:00:46.787961 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":61,"threads_started":1}
Sep 26 04:00:47 dist-test-slave-nhpw krb5kdc[12145](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790395247, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:00:47.420585 13339 heartbeater.cc:464] Registering TS with master...
I20260926 04:00:47.420780 13339 heartbeater.cc:511] Master 127.0.98.62:44347 requested a full tablet report, sending...
I20260926 04:00:47.421188 13141 ts_manager.cc:194] Registered new tserver with Master: 9a11c2f492b1455a9369a077ac1536aa (127.0.98.1:32821)
I20260926 04:00:47.421871 13141 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56871
I20260926 04:00:47.597149 13473 heartbeater.cc:464] Registering TS with master...
I20260926 04:00:47.597314 13473 heartbeater.cc:511] Master 127.0.98.62:44347 requested a full tablet report, sending...
I20260926 04:00:47.597625 13141 ts_manager.cc:194] Registered new tserver with Master: cd33bc3d9b054802a4469efc639afa35 (127.0.98.2:46447)
I20260926 04:00:47.598160 13141 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54551
Sep 26 04:00:47 dist-test-slave-nhpw krb5kdc[12145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395247, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260926 04:00:47.786286 13607 heartbeater.cc:464] Registering TS with master...
I20260926 04:00:47.786473 13607 heartbeater.cc:511] Master 127.0.98.62:44347 requested a full tablet report, sending...
I20260926 04:00:47.786787 13141 ts_manager.cc:194] Registered new tserver with Master: ff5b3d52cc2a4d93904265fedf382e36 (127.0.98.3:35537)
I20260926 04:00:47.787191 13141 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46201
I20260926 04:00:47.796470 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 04:00:47.804129 13629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:47.797501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49016 (local address 127.0.98.62:44347)
0926 04:00:47.797615 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 04:00:47.797628 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 04:00:47.797664 (+ 36us) server_negotiation.cc:407] Connection header received
0926 04:00:47.797731 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:00:47.797732 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:00:47.797745 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:00:47.797808 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:00:47.798387 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:47.798923 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:47.799691 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:47.799910 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:47.799993 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:47.801343 (+ 1350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:00:47.801344 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:00:47.801345 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:00:47.801352 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:00:47.802984 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:00:47.803337 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:00:47.803338 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:00:47.803338 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:00:47.803401 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:00:47.803602 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:00:47.803603 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:00:47.803603 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:00:47.803725 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:00:47.803834 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 04:00:47.803928 (+ 94us) server_negotiation.cc:299] Negotiation successful
0926 04:00:47.804008 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":66,"threads_started":1}
I20260926 04:00:47.806124 13141 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49016:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 04:00:47.807801 13141 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 04:00:47.815526 13635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:47.810524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46447 (local address 127.0.0.1:45336)
0926 04:00:47.811168 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0926 04:00:47.811177 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 04:00:47.811237 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:00:47.811513 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:00:47.811516 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:00:47.811518 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:00:47.811581 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:00:47.811583 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:00:47.812872 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:00:47.812873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:00:47.814195 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:00:47.814198 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:00:47.815217 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:00:47.815218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:00:47.815297 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:47.815311 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 04:00:47.815376 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 04:00:47.815431 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":627,"mutex_wait_us":4,"thread_start_us":76,"threads_started":1}
I20260926 04:00:47.815783 13634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:47.810524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35537 (local address 127.0.0.1:60532)
0926 04:00:47.810965 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0926 04:00:47.811008 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 04:00:47.811573 (+ 565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:00:47.811839 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:00:47.811840 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:00:47.811842 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:00:47.811874 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:00:47.811876 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:00:47.813339 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:00:47.813341 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:00:47.814360 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:00:47.814362 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:00:47.815553 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:00:47.815554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:00:47.815634 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:47.815638 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 04:00:47.815686 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 04:00:47.815719 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":82,"threads_started":1}
I20260926 04:00:47.816154 13638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:47.810768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45336 (local address 127.0.98.2:46447)
0926 04:00:47.810905 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 04:00:47.810910 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 04:00:47.811325 (+ 415us) server_negotiation.cc:407] Connection header received
0926 04:00:47.811386 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:00:47.811388 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:00:47.811404 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:00:47.811498 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:00:47.811661 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:47.812785 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:47.814289 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:47.815015 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:47.815088 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:47.815181 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 04:00:47.816020 (+ 839us) server_negotiation.cc:299] Negotiation successful
0926 04:00:47.816070 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":51,"threads_started":1}
I20260926 04:00:47.816193 13639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:47.810768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60532 (local address 127.0.98.3:35537)
0926 04:00:47.810905 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 04:00:47.810910 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 04:00:47.811080 (+ 170us) server_negotiation.cc:407] Connection header received
0926 04:00:47.811683 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:00:47.811687 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:00:47.811705 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:00:47.811771 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:00:47.811945 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:47.813261 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:47.814441 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:47.815118 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:47.815183 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:47.815522 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0926 04:00:47.816070 (+ 548us) server_negotiation.cc:299] Negotiation successful
0926 04:00:47.816115 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":60,"threads_started":1}
I20260926 04:00:47.816916 13541 tablet_service.cc:1467] Processing CreateTablet for tablet 7e915138b3fc4ee1911a23043736ec49 (DEFAULT_TABLE table=test-table [id=b3b603a346e04520922d82ce52a6ec0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:00:47.817036 13407 tablet_service.cc:1467] Processing CreateTablet for tablet 7e915138b3fc4ee1911a23043736ec49 (DEFAULT_TABLE table=test-table [id=b3b603a346e04520922d82ce52a6ec0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:00:47.817294 13541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e915138b3fc4ee1911a23043736ec49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:00:47.817436 13407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e915138b3fc4ee1911a23043736ec49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:00:47.819976 13640 tablet_bootstrap.cc:492] T 7e915138b3fc4ee1911a23043736ec49 P ff5b3d52cc2a4d93904265fedf382e36: Bootstrap starting.
I20260926 04:00:47.820719 13640 tablet_bootstrap.cc:654] T 7e915138b3fc4ee1911a23043736ec49 P ff5b3d52cc2a4d93904265fedf382e36: Neither blocks nor log segments found. Creating new log.
I20260926 04:00:47.821266 13640 log.cc:824] T 7e915138b3fc4ee1911a23043736ec49 P ff5b3d52cc2a4d93904265fedf382e36: Log is configured to *not* fsync() on all Append() calls
I20260926 04:00:47.823249 13640 tablet_bootstrap.cc:492] T 7e915138b3fc4ee1911a23043736ec49 P ff5b3d52cc2a4d93904265fedf382e36: No bootstrap required, opened a new log
I20260926 04:00:47.823382 13640 ts_tablet_manager.cc:1397] T 7e915138b3fc4ee1911a23043736ec49 P ff5b3d52cc2a4d93904265fedf382e36: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260926 04:00:47.824290 13635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:47.810541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32821 (local address 127.0.0.1:35334)
0926 04:00:47.815909 (+ 5368us) negotiation.cc:107] Waiting for socket to connect
0926 04:00:47.815919 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 04:00:47.815950 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:00:47.820938 (+ 4988us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:00:47.820939 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:00:47.820941 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:00:47.820972 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:00:47.820973 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:00:47.822086 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:00:47.822087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:00:47.823133 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:00:47.823136 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:00:47.824076 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:00:47.824077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:00:47.824144 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:47.824147 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 04:00:47.824193 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 04:00:47.824229 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5358,"thread_start_us":105,"threads_started":1}
I20260926 04:00:47.824570 13637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:47.810637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35334 (local address 127.0.98.1:32821)
0926 04:00:47.820782 (+ 10145us) server_negotiation.cc:206] Beginning negotiation
0926 04:00:47.820787 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 04:00:47.820799 (+ 12us) server_negotiation.cc:407] Connection header received
0926 04:00:47.820834 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:00:47.820836 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:00:47.820852 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:00:47.820944 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:00:47.821025 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:47.822001 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:47.823235 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:47.823912 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:47.823992 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:47.824045 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 04:00:47.824452 (+ 407us) server_negotiation.cc:299] Negotiation successful
0926 04:00:47.824498 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10104,"thread_start_us":39,"threads_started":1}
I20260926 04:00:47.825271 13273 tablet_service.cc:1467] Processing CreateTablet for tablet 7e915138b3fc4ee1911a23043736ec49 (DEFAULT_TABLE table=test-table [id=b3b603a346e04520922d82ce52a6ec0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:00:47.825562 13273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7e915138b3fc4ee1911a23043736ec49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:00:47.826402 13640 raft_consensus.cc:348] T 7e915138b3fc4ee1911a23043736ec49 P ff5b3d52cc2a4d93904265fedf382e36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd33bc3d9b054802a4469efc639afa35" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46447 } } peers { permanent_uuid: "ff5b3d52cc2a4d93904265fedf382e36" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35537 } } peers { permanent_uuid: "9a11c2f492b1455a9369a077ac1536aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32821 } }
I20260926 04:00:47.826567 13640 raft_consensus.cc:374] T 7e915138b3fc4ee1911a23043736ec49 P ff5b3d52cc2a4d93904265fedf382e36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:00:47.826603 13640 raft_consensus.cc:729] T 7e915138b3fc4ee1911a23043736ec49 P ff5b3d52cc2a4d93904265fedf382e36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff5b3d52cc2a4d93904265fedf382e36, State: Initialized, Role: FOLLOWER
I20260926 04:00:47.826733 13640 consensus_queue.cc:260] T 7e915138b3fc4ee1911a23043736ec49 P ff5b3d52cc2a4d93904265fedf382e36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd33bc3d9b054802a4469efc639afa35" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46447 } } peers { permanent_uuid: "ff5b3d52cc2a4d93904265fedf382e36" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35537 } } peers { permanent_uuid: "9a11c2f492b1455a9369a077ac1536aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32821 } }
I20260926 04:00:47.826996 13640 ts_tablet_manager.cc:1428] T 7e915138b3fc4ee1911a23043736ec49 P ff5b3d52cc2a4d93904265fedf382e36: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260926 04:00:47.827106 13607 heartbeater.cc:503] Master 127.0.98.62:44347 was elected leader, sending a full tablet report...
I20260926 04:00:47.828876 13644 tablet_bootstrap.cc:492] T 7e915138b3fc4ee1911a23043736ec49 P 9a11c2f492b1455a9369a077ac1536aa: Bootstrap starting.
I20260926 04:00:47.829480 13644 tablet_bootstrap.cc:654] T 7e915138b3fc4ee1911a23043736ec49 P 9a11c2f492b1455a9369a077ac1536aa: Neither blocks nor log segments found. Creating new log.
I20260926 04:00:47.829775 13644 log.cc:824] T 7e915138b3fc4ee1911a23043736ec49 P 9a11c2f492b1455a9369a077ac1536aa: Log is configured to *not* fsync() on all Append() calls
I20260926 04:00:47.830506 13644 tablet_bootstrap.cc:492] T 7e915138b3fc4ee1911a23043736ec49 P 9a11c2f492b1455a9369a077ac1536aa: No bootstrap required, opened a new log
I20260926 04:00:47.830571 13644 ts_tablet_manager.cc:1397] T 7e915138b3fc4ee1911a23043736ec49 P 9a11c2f492b1455a9369a077ac1536aa: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 04:00:47.832762 13641 tablet_bootstrap.cc:492] T 7e915138b3fc4ee1911a23043736ec49 P cd33bc3d9b054802a4469efc639afa35: Bootstrap starting.
I20260926 04:00:47.832953 13644 raft_consensus.cc:348] T 7e915138b3fc4ee1911a23043736ec49 P 9a11c2f492b1455a9369a077ac1536aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd33bc3d9b054802a4469efc639afa35" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46447 } } peers { permanent_uuid: "ff5b3d52cc2a4d93904265fedf382e36" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35537 } } peers { permanent_uuid: "9a11c2f492b1455a9369a077ac1536aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32821 } }
I20260926 04:00:47.833125 13644 raft_consensus.cc:374] T 7e915138b3fc4ee1911a23043736ec49 P 9a11c2f492b1455a9369a077ac1536aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:00:47.833158 13644 raft_consensus.cc:729] T 7e915138b3fc4ee1911a23043736ec49 P 9a11c2f492b1455a9369a077ac1536aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a11c2f492b1455a9369a077ac1536aa, State: Initialized, Role: FOLLOWER
I20260926 04:00:47.833267 13644 consensus_queue.cc:260] T 7e915138b3fc4ee1911a23043736ec49 P 9a11c2f492b1455a9369a077ac1536aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd33bc3d9b054802a4469efc639afa35" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46447 } } peers { permanent_uuid: "ff5b3d52cc2a4d93904265fedf382e36" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35537 } } peers { permanent_uuid: "9a11c2f492b1455a9369a077ac1536aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32821 } }
I20260926 04:00:47.833402 13641 tablet_bootstrap.cc:654] T 7e915138b3fc4ee1911a23043736ec49 P cd33bc3d9b054802a4469efc639afa35: Neither blocks nor log segments found. Creating new log.
I20260926 04:00:47.833515 13644 ts_tablet_manager.cc:1428] T 7e915138b3fc4ee1911a23043736ec49 P 9a11c2f492b1455a9369a077ac1536aa: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 04:00:47.833666 13339 heartbeater.cc:503] Master 127.0.98.62:44347 was elected leader, sending a full tablet report...
I20260926 04:00:47.833721 13641 log.cc:824] T 7e915138b3fc4ee1911a23043736ec49 P cd33bc3d9b054802a4469efc639afa35: Log is configured to *not* fsync() on all Append() calls
I20260926 04:00:47.834400 13641 tablet_bootstrap.cc:492] T 7e915138b3fc4ee1911a23043736ec49 P cd33bc3d9b054802a4469efc639afa35: No bootstrap required, opened a new log
I20260926 04:00:47.834475 13641 ts_tablet_manager.cc:1397] T 7e915138b3fc4ee1911a23043736ec49 P cd33bc3d9b054802a4469efc639afa35: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 04:00:47.836338 13641 raft_consensus.cc:348] T 7e915138b3fc4ee1911a23043736ec49 P cd33bc3d9b054802a4469efc639afa35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd33bc3d9b054802a4469efc639afa35" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46447 } } peers { permanent_uuid: "ff5b3d52cc2a4d93904265fedf382e36" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35537 } } peers { permanent_uuid: "9a11c2f492b1455a9369a077ac1536aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32821 } }
I20260926 04:00:47.836452 13641 raft_consensus.cc:374] T 7e915138b3fc4ee1911a23043736ec49 P cd33bc3d9b054802a4469efc639afa35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:00:47.836473 13641 raft_consensus.cc:729] T 7e915138b3fc4ee1911a23043736ec49 P cd33bc3d9b054802a4469efc639afa35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd33bc3d9b054802a4469efc639afa35, State: Initialized, Role: FOLLOWER
I20260926 04:00:47.836582 13641 consensus_queue.cc:260] T 7e915138b3fc4ee1911a23043736ec49 P cd33bc3d9b054802a4469efc639afa35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd33bc3d9b054802a4469efc639afa35" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46447 } } peers { permanent_uuid: "ff5b3d52cc2a4d93904265fedf382e36" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35537 } } peers { permanent_uuid: "9a11c2f492b1455a9369a077ac1536aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32821 } }
I20260926 04:00:47.836838 13641 ts_tablet_manager.cc:1428] T 7e915138b3fc4ee1911a23043736ec49 P cd33bc3d9b054802a4469efc639afa35: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 04:00:47.836891 13473 heartbeater.cc:503] Master 127.0.98.62:44347 was elected leader, sending a full tablet report...
I20260926 04:00:47.875872 13643 raft_consensus.cc:482] T 7e915138b3fc4ee1911a23043736ec49 P ff5b3d52cc2a4d93904265fedf382e36 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 04:00:47.876026 13643 raft_consensus.cc:504] T 7e915138b3fc4ee1911a23043736ec49 P ff5b3d52cc2a4d93904265fedf382e36 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd33bc3d9b054802a4469efc639afa35" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46447 } } peers { permanent_uuid: "ff5b3d52cc2a4d93904265fedf382e36" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35537 } } peers { permanent_uuid: "9a11c2f492b1455a9369a077ac1536aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32821 } }
I20260926 04:00:47.876430 13643 leader_election.cc:288] T 7e915138b3fc4ee1911a23043736ec49 P ff5b3d52cc2a4d93904265fedf382e36 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd33bc3d9b054802a4469efc639afa35 (127.0.98.2:46447), 9a11c2f492b1455a9369a077ac1536aa (127.0.98.1:32821)
I20260926 04:00:47.880512 13637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:47.877073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52743 (local address 127.0.98.1:32821)
0926 04:00:47.877120 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 04:00:47.877122 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:00:47.877248 (+ 126us) server_negotiation.cc:407] Connection header received
0926 04:00:47.877347 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:00:47.877348 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:00:47.877358 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:00:47.877385 (+ 27us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:00:47.877585 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:47.878264 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:47.879603 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:47.880143 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:47.880188 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:47.880198 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 04:00:47.880401 (+ 203us) server_negotiation.cc:299] Negotiation successful
0926 04:00:47.880442 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 04:00:47.880656 13638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:47.876926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60151 (local address 127.0.98.2:46447)
0926 04:00:47.876976 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 04:00:47.876979 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:00:47.876988 (+ 9us) server_negotiation.cc:407] Connection header received
0926 04:00:47.877132 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:00:47.877134 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:00:47.877146 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:00:47.877178 (+ 32us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:00:47.877450 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:47.878191 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:47.879625 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:47.880209 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:47.880258 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:47.880267 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 04:00:47.880526 (+ 259us) server_negotiation.cc:299] Negotiation successful
0926 04:00:47.880571 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 04:00:47.880903 13649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:47.876604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46447 (local address 127.0.98.3:60151)
0926 04:00:47.876884 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0926 04:00:47.876896 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 04:00:47.877016 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:00:47.877258 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:00:47.877259 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:00:47.877261 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:00:47.877358 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:00:47.877361 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:00:47.878292 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:00:47.878293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:00:47.879518 (+ 1225us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:00:47.879521 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:00:47.880346 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:00:47.880347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:00:47.880432 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:47.880436 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 04:00:47.880772 (+ 336us) client_negotiation.cc:240] Negotiation successful
0926 04:00:47.880815 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":142,"threads_started":1}
I20260926 04:00:47.881259 13650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:47.876725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32821 (local address 127.0.98.3:52743)
0926 04:00:47.877181 (+ 456us) negotiation.cc:107] Waiting for socket to connect
0926 04:00:47.877191 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 04:00:47.877273 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:00:47.877445 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:00:47.877445 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:00:47.877447 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:00:47.877517 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:00:47.877519 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:00:47.878364 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:00:47.878364 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:00:47.879503 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:00:47.879505 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:00:47.880254 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:00:47.880255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:00:47.880321 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:47.880325 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 04:00:47.881142 (+ 817us) client_negotiation.cc:240] Negotiation successful
0926 04:00:47.881178 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":94,"threads_started":1}
I20260926 04:00:47.881202 13427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e915138b3fc4ee1911a23043736ec49" candidate_uuid: "ff5b3d52cc2a4d93904265fedf382e36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd33bc3d9b054802a4469efc639afa35" is_pre_election: true
I20260926 04:00:47.881366 13427 raft_consensus.cc:2454] T 7e915138b3fc4ee1911a23043736ec49 P cd33bc3d9b054802a4469efc639afa35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff5b3d52cc2a4d93904265fedf382e36 in term 0.
I20260926 04:00:47.881608 13495 leader_election.cc:302] T 7e915138b3fc4ee1911a23043736ec49 P ff5b3d52cc2a4d93904265fedf382e36 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cd33bc3d9b054802a4469efc639afa35, ff5b3d52cc2a4d93904265fedf382e36; no voters:
I20260926 04:00:47.881608 13292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e915138b3fc4ee1911a23043736ec49" candidate_uuid: "ff5b3d52cc2a4d93904265fedf382e36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a11c2f492b1455a9369a077ac1536aa" is_pre_election: true
I20260926 04:00:47.881759 13292 raft_consensus.cc:2454] T 7e915138b3fc4ee1911a23043736ec49 P 9a11c2f492b1455a9369a077ac1536aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff5b3d52cc2a4d93904265fedf382e36 in term 0.
I20260926 04:00:47.881758 13643 raft_consensus.cc:2781] T 7e915138b3fc4ee1911a23043736ec49 P ff5b3d52cc2a4d93904265fedf382e36 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 04:00:47.881801 13643 raft_consensus.cc:482] T 7e915138b3fc4ee1911a23043736ec49 P ff5b3d52cc2a4d93904265fedf382e36 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 04:00:47.881824 13643 raft_consensus.cc:3037] T 7e915138b3fc4ee1911a23043736ec49 P ff5b3d52cc2a4d93904265fedf382e36 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:00:47.882681 13643 raft_consensus.cc:504] T 7e915138b3fc4ee1911a23043736ec49 P ff5b3d52cc2a4d93904265fedf382e36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd33bc3d9b054802a4469efc639afa35" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46447 } } peers { permanent_uuid: "ff5b3d52cc2a4d93904265fedf382e36" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35537 } } peers { permanent_uuid: "9a11c2f492b1455a9369a077ac1536aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32821 } }
I20260926 04:00:47.882812 13643 leader_election.cc:288] T 7e915138b3fc4ee1911a23043736ec49 P ff5b3d52cc2a4d93904265fedf382e36 [CANDIDATE]: Term 1 election: Requested vote from peers cd33bc3d9b054802a4469efc639afa35 (127.0.98.2:46447), 9a11c2f492b1455a9369a077ac1536aa (127.0.98.1:32821)
I20260926 04:00:47.883070 13292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e915138b3fc4ee1911a23043736ec49" candidate_uuid: "ff5b3d52cc2a4d93904265fedf382e36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a11c2f492b1455a9369a077ac1536aa"
I20260926 04:00:47.883139 13292 raft_consensus.cc:3037] T 7e915138b3fc4ee1911a23043736ec49 P 9a11c2f492b1455a9369a077ac1536aa [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:00:47.883145 13427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7e915138b3fc4ee1911a23043736ec49" candidate_uuid: "ff5b3d52cc2a4d93904265fedf382e36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd33bc3d9b054802a4469efc639afa35"
I20260926 04:00:47.883226 13427 raft_consensus.cc:3037] T 7e915138b3fc4ee1911a23043736ec49 P cd33bc3d9b054802a4469efc639afa35 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:00:47.883961 13292 raft_consensus.cc:2454] T 7e915138b3fc4ee1911a23043736ec49 P 9a11c2f492b1455a9369a077ac1536aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff5b3d52cc2a4d93904265fedf382e36 in term 1.
I20260926 04:00:47.884130 13427 raft_consensus.cc:2454] T 7e915138b3fc4ee1911a23043736ec49 P cd33bc3d9b054802a4469efc639afa35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff5b3d52cc2a4d93904265fedf382e36 in term 1.
I20260926 04:00:47.884423 13496 leader_election.cc:302] T 7e915138b3fc4ee1911a23043736ec49 P ff5b3d52cc2a4d93904265fedf382e36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9a11c2f492b1455a9369a077ac1536aa, ff5b3d52cc2a4d93904265fedf382e36; no voters:
I20260926 04:00:47.884507 13643 raft_consensus.cc:2781] T 7e915138b3fc4ee1911a23043736ec49 P ff5b3d52cc2a4d93904265fedf382e36 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:00:47.884690 13643 raft_consensus.cc:686] T 7e915138b3fc4ee1911a23043736ec49 P ff5b3d52cc2a4d93904265fedf382e36 [term 1 LEADER]: Becoming Leader. State: Replica: ff5b3d52cc2a4d93904265fedf382e36, State: Running, Role: LEADER
I20260926 04:00:47.884788 13643 consensus_queue.cc:237] T 7e915138b3fc4ee1911a23043736ec49 P ff5b3d52cc2a4d93904265fedf382e36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd33bc3d9b054802a4469efc639afa35" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46447 } } peers { permanent_uuid: "ff5b3d52cc2a4d93904265fedf382e36" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35537 } } peers { permanent_uuid: "9a11c2f492b1455a9369a077ac1536aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32821 } }
I20260926 04:00:47.885492 13141 catalog_manager.cc:5495] T 7e915138b3fc4ee1911a23043736ec49 P ff5b3d52cc2a4d93904265fedf382e36 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff5b3d52cc2a4d93904265fedf382e36 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "ff5b3d52cc2a4d93904265fedf382e36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd33bc3d9b054802a4469efc639afa35" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff5b3d52cc2a4d93904265fedf382e36" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35537 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9a11c2f492b1455a9369a077ac1536aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32821 } health_report { overall_health: UNKNOWN } } }
W20260926 04:00:47.895830 13340 tablet.cc:2369] T 7e915138b3fc4ee1911a23043736ec49 P 9a11c2f492b1455a9369a077ac1536aa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 04:00:47.908098 13639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:47.905117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60536 (local address 127.0.98.3:35537)
0926 04:00:47.905157 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 04:00:47.905159 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:00:47.905168 (+ 9us) server_negotiation.cc:407] Connection header received
0926 04:00:47.905232 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:00:47.905233 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:00:47.905254 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:00:47.905328 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 04:00:47.905476 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:47.906168 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:47.907228 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:47.907497 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:47.907561 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:47.907738 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 04:00:47.907865 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 04:00:47.907943 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 04:00:47.908006 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 04:00:47.908027 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 04:00:47.910085 13292 raft_consensus.cc:1261] T 7e915138b3fc4ee1911a23043736ec49 P 9a11c2f492b1455a9369a077ac1536aa [term 1 FOLLOWER]: Refusing update from remote peer ff5b3d52cc2a4d93904265fedf382e36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:00:47.910079 13427 raft_consensus.cc:1261] T 7e915138b3fc4ee1911a23043736ec49 P cd33bc3d9b054802a4469efc639afa35 [term 1 FOLLOWER]: Refusing update from remote peer ff5b3d52cc2a4d93904265fedf382e36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:00:47.910414 13651 consensus_queue.cc:1035] T 7e915138b3fc4ee1911a23043736ec49 P ff5b3d52cc2a4d93904265fedf382e36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd33bc3d9b054802a4469efc639afa35" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:00:47.910521 13643 consensus_queue.cc:1035] T 7e915138b3fc4ee1911a23043736ec49 P ff5b3d52cc2a4d93904265fedf382e36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a11c2f492b1455a9369a077ac1536aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 32821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:00:47.914690 13654 mvcc.cc:205] Tried to move back new op lower bound from 7333458935436509184 to 7333458935336566784. Current Snapshot: MvccSnapshot[applied={T|T < 7333458935436509184}]
I20260926 04:00:47.916342 13657 mvcc.cc:205] Tried to move back new op lower bound from 7333458935436509184 to 7333458935336566784. Current Snapshot: MvccSnapshot[applied={T|T < 7333458935436509184}]
I20260926 04:00:47.918449 13655 mvcc.cc:205] Tried to move back new op lower bound from 7333458935436509184 to 7333458935336566784. Current Snapshot: MvccSnapshot[applied={T|T < 7333458935436509184}]
I20260926 04:00:47.919375 13141 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49016:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 04:00:47.919476 13141 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49016:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 04:00:47.920217 13141 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 078e9a0c301348fdaa2792803826dae9: Sending DeleteTablet for 3 replicas of tablet 7e915138b3fc4ee1911a23043736ec49
I20260926 04:00:47.920536 13273 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e915138b3fc4ee1911a23043736ec49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:00:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35334
I20260926 04:00:47.920580 13541 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e915138b3fc4ee1911a23043736ec49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:00:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60532
I20260926 04:00:47.920786 13407 tablet_service.cc:1514] Processing DeleteTablet for tablet 7e915138b3fc4ee1911a23043736ec49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:00:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45336
I20260926 04:00:47.920984 13665 tablet_replica.cc:330] T 7e915138b3fc4ee1911a23043736ec49 P cd33bc3d9b054802a4469efc639afa35: stopping tablet replica
I20260926 04:00:47.921075 13665 raft_consensus.cc:2229] T 7e915138b3fc4ee1911a23043736ec49 P cd33bc3d9b054802a4469efc639afa35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 04:00:47.921211 13665 raft_consensus.cc:2258] T 7e915138b3fc4ee1911a23043736ec49 P cd33bc3d9b054802a4469efc639afa35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:00:47.921258 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 13180
I20260926 04:00:47.922551 13665 ts_tablet_manager.cc:1905] T 7e915138b3fc4ee1911a23043736ec49 P cd33bc3d9b054802a4469efc639afa35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:00:47.923522 13664 tablet_replica.cc:330] T 7e915138b3fc4ee1911a23043736ec49 P ff5b3d52cc2a4d93904265fedf382e36: stopping tablet replica
I20260926 04:00:47.923678 13664 raft_consensus.cc:2229] T 7e915138b3fc4ee1911a23043736ec49 P ff5b3d52cc2a4d93904265fedf382e36 [term 1 LEADER]: Raft consensus shutting down.
I20260926 04:00:47.923810 13664 raft_consensus.cc:2258] T 7e915138b3fc4ee1911a23043736ec49 P ff5b3d52cc2a4d93904265fedf382e36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:00:47.924278 13664 ts_tablet_manager.cc:1905] T 7e915138b3fc4ee1911a23043736ec49 P ff5b3d52cc2a4d93904265fedf382e36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:00:47.924353 13665 ts_tablet_manager.cc:1918] T 7e915138b3fc4ee1911a23043736ec49 P cd33bc3d9b054802a4469efc639afa35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 04:00:47.924431 13665 log.cc:1196] T 7e915138b3fc4ee1911a23043736ec49 P cd33bc3d9b054802a4469efc639afa35: Deleting WAL directory at /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-1/wal/wals/7e915138b3fc4ee1911a23043736ec49
I20260926 04:00:47.924752 13665 ts_tablet_manager.cc:1939] T 7e915138b3fc4ee1911a23043736ec49 P cd33bc3d9b054802a4469efc639afa35: Deleting consensus metadata
I20260926 04:00:47.925163 13126 catalog_manager.cc:4841] TS cd33bc3d9b054802a4469efc639afa35 (127.0.98.2:46447): tablet 7e915138b3fc4ee1911a23043736ec49 (table test-table [id=b3b603a346e04520922d82ce52a6ec0c]) successfully deleted
I20260926 04:00:47.926172 13664 ts_tablet_manager.cc:1918] T 7e915138b3fc4ee1911a23043736ec49 P ff5b3d52cc2a4d93904265fedf382e36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 04:00:47.926251 13664 log.cc:1196] T 7e915138b3fc4ee1911a23043736ec49 P ff5b3d52cc2a4d93904265fedf382e36: Deleting WAL directory at /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790395095474643-392-0/minicluster-data/ts-2/wal/wals/7e915138b3fc4ee1911a23043736ec49
I20260926 04:00:47.926542 13664 ts_tablet_manager.cc:1939] T 7e915138b3fc4ee1911a23043736ec49 P ff5b3d52cc2a4d93904265fedf382e36: Deleting consensus metadata
I20260926 04:00:47.926929 13128 catalog_manager.cc:4841] TS ff5b3d52cc2a4d93904265fedf382e36 (127.0.98.3:35537): tablet 7e915138b3fc4ee1911a23043736ec49 (table test-table [id=b3b603a346e04520922d82ce52a6ec0c]) successfully deleted
W20260926 04:00:47.929651 13127 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:32821 (error 108)
W20260926 04:00:47.930292 13635 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:00:47.930057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:32821 (local address 127.0.0.1:35342)
0926 04:00:47.930124 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0926 04:00:47.930138 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 04:00:47.930199 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:00:47.930245 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:32821: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 04:00:47.930495 13127 catalog_manager.cc:4584] TS 9a11c2f492b1455a9369a077ac1536aa (127.0.98.1:32821): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7e915138b3fc4ee1911a23043736ec49: Network error: Client connection negotiation failed: client connection to 127.0.98.1:32821: BlockingWrite error: write error: Connection reset by peer (error 104)
I20260926 04:00:47.930794 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 13343
I20260926 04:00:47.935456 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 13476
I20260926 04:00:47.941123 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 13108
I20260926 04:00:47.946789 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 04:00:48.291028 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 04:00:48.291121 392 mini_ranger.cc:66] Stopping Ranger...
I20260926 04:00:48.635355 392 mini_ranger.cc:68] Stopped Ranger
2026-09-26 04:00:48.635 UTC [12175] LOG: received smart shutdown request
2026-09-26 04:00:48.639 UTC [12175] LOG: background worker "logical replication launcher" (PID 12184) exited with exit code 1
2026-09-26 04:00:48.639 UTC [12179] LOG: shutting down
2026-09-26 04:00:48.811 UTC [12175] LOG: database system is shut down
2026-09-26T04:00:48Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48200 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395095474643-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 04:00:48 dist-test-slave-nhpw krb5kdc[13676](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 04:00:48 dist-test-slave-nhpw krb5kdc[13676](info): set up 2 sockets
Sep 26 04:00:48 dist-test-slave-nhpw krb5kdc[13676](info): commencing operation
krb5kdc: starting...
W20260926 04:00:49.948451 392 mini_kdc.cc:121] Time spent starting KDC: real 1.008s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395095474643-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395095474643-392-0/krb5kdc/test-user.keytab.
Sep 26 04:00:50 dist-test-slave-nhpw krb5kdc[13676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T04:00:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:00:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 04:00:50.068617 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395095474643-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395095474643-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395095474643-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37273
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37273,127.0.98.61:36457,127.0.98.60:45123
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:00:50.148401 13689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:00:50.148633 13689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:00:50.148692 13689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:00:50.149802 13689 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:00:50.149871 13689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:00:50.149897 13689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:00:50.149928 13689 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:00:50.149953 13689 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:00:50.151424 13689 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395095474643-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395095474643-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37273,127.0.98.61:36457,127.0.98.60:45123
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395095474643-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:00:50.151688 13689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:00:50.151899 13689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:00:50.156625 13697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:00:50.156625 13695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:00:50.156807 13689 server_base.cc:1034] running on GCE node
W20260926 04:00:50.156720 13694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:00:50.157131 13689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:00:50.157411 13689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:00:50.158576 13689 hybrid_clock.cc:648] HybridClock initialized: now 1790395250158568 us; error 46 us; skew 500 ppm
Sep 26 04:00:50 dist-test-slave-nhpw krb5kdc[13676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395250, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:00:50.164290 13689 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 04:00:50.164795 13689 webserver.cc:460] Webserver started at http://127.0.98.62:37223/ using document root <none> and password file <none>
I20260926 04:00:50.165042 13689 fs_manager.cc:359] Metadata directory not provided
I20260926 04:00:50.165140 13689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:00:50.165287 13689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:00:50.166442 13689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395095474643-392-0/minicluster-data/master-0/data/instance:
uuid: "69a8e46547974380b7c57d002f309c19"
format_stamp: "Formatted at 2026-09-26 04:00:50 on dist-test-slave-nhpw"
server_key: "66c3be848b028736c6e61c2705a402ff"
server_key_iv: "82d7ec1fd23d0a89a4b4290eed6c0a33"
server_key_version: "encryptionkey@0"
I20260926 04:00:50.166890 13689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395095474643-392-0/minicluster-data/master-0/wal/instance:
uuid: "69a8e46547974380b7c57d002f309c19"
format_stamp: "Formatted at 2026-09-26 04:00:50 on dist-test-slave-nhpw"
server_key: "66c3be848b028736c6e61c2705a402ff"
server_key_iv: "82d7ec1fd23d0a89a4b4290eed6c0a33"
server_key_version: "encryptionkey@0"
I20260926 04:00:50.168217 13689 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 04:00:50.168920 13704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:00:50.169153 13689 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:00:50.169234 13689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395095474643-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395095474643-392-0/minicluster-data/master-0/wal
uuid: "69a8e46547974380b7c57d002f309c19"
format_stamp: "Formatted at 2026-09-26 04:00:50 on dist-test-slave-nhpw"
server_key: "66c3be848b028736c6e61c2705a402ff"
server_key_iv: "82d7ec1fd23d0a89a4b4290eed6c0a33"
server_key_version: "encryptionkey@0"
I20260926 04:00:50.169287 13689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395095474643-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395095474643-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395095474643-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:00:50.178424 13689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:00:50.180079 13689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:00:50.180189 13689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:00:50.184172 13689 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37273
I20260926 04:00:50.184202 13756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37273 every 8 connection(s)
I20260926 04:00:50.184540 13689 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
I20260926 04:00:50.185421 13757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:00:50.186916 13757 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37273 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 04:00:50 dist-test-slave-nhpw krb5kdc[13676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395250, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 04:00:50.192898 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 13689
I20260926 04:00:50.192994 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395095474643-392-0/minicluster-data/master-0/wal/instance
I20260926 04:00:50.193168 392 external_mini_cluster.cc:1428] Setting key 4ce994aea128ad1ceccc360d2f8e28d5
I20260926 04:00:50.196630 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:50.187619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55366 (local address 127.0.98.62:37273)
0926 04:00:50.187807 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 04:00:50.187811 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 04:00:50.187863 (+ 52us) server_negotiation.cc:407] Connection header received
0926 04:00:50.188434 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:00:50.188436 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:00:50.188453 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:00:50.188531 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:00:50.189028 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:50.189840 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:50.190706 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:50.191103 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:50.191187 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:50.193448 (+ 2261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:00:50.193455 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:00:50.193456 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:00:50.193469 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:00:50.195001 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:00:50.195430 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:00:50.195432 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:00:50.195432 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:00:50.195512 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:00:50.195770 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:00:50.195771 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:00:50.195771 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:00:50.195977 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:00:50.196082 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 04:00:50.196271 (+ 189us) server_negotiation.cc:299] Negotiation successful
0926 04:00:50.196400 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":124,"threads_started":1}
I20260926 04:00:50.196659 13759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:50.187588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37273 (local address 127.0.0.1:55366)
0926 04:00:50.187752 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 04:00:50.187791 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 04:00:50.188265 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:00:50.188543 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:00:50.188545 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:00:50.188802 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:00:50.188948 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:00:50.188952 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:00:50.189922 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:00:50.189923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:00:50.190580 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:00:50.190581 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:00:50.191182 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:00:50.191183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:00:50.191242 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:50.191760 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:00:50.191768 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:00:50.193349 (+ 1581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:00:50.195127 (+ 1778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:00:50.195132 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:00:50.195134 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:00:50.195330 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:00:50.195602 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:00:50.195603 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:00:50.195603 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:00:50.195702 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:00:50.196065 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:00:50.196069 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:00:50.196129 (+ 60us) client_negotiation.cc:769] Sending connection context
0926 04:00:50.196234 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 04:00:50.196462 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":96,"thread_start_us":89,"threads_started":1}
I20260926 04:00:50.197424 13757 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36457 } has no permanent_uuid. Determining permanent_uuid...
I20260926 04:00:50.197767 13759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:50.197651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36457 (local address 127.0.0.1:57682)
0926 04:00:50.197696 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 04:00:50.197736 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 04:00:50.197923 13705 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36457: connect: Connection refused (error 111)
I20260926 04:00:50.198318 13759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:50.198226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36457 (local address 127.0.0.1:57686)
0926 04:00:50.198274 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 04:00:50.198298 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 04:00:50.198426 13757 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36457: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36457: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395095474643-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395095474643-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260926 04:00:50.219583 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395095474643-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395095474643-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395095474643-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395095474643-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:36457
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395095474643-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37273,127.0.98.61:36457,127.0.98.60:45123
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 04:00:50.247689 13757 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36457 } attempt: 1
I20260926 04:00:50.248238 13759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:50.248077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36457 (local address 127.0.0.1:57702)
0926 04:00:50.248155 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0926 04:00:50.248205 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260926 04:00:50.248615 13759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:50.248509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36457 (local address 127.0.0.1:57712)
0926 04:00:50.248562 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 04:00:50.248588 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 04:00:50.248747 13757 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36457: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36457: connect: Connection refused (error 111)
W20260926 04:00:50.301900 13764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:00:50.302187 13764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:00:50.302225 13764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:00:50.303942 13764 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:00:50.303983 13764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:00:50.303997 13764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:00:50.304042 13764 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:00:50.304069 13764 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:00:50.305764 13764 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395095474643-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395095474643-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37273,127.0.98.61:36457,127.0.98.60:45123
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395095474643-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:36457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395095474643-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13764
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395095474643-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:00:50.306005 13764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:00:50.306249 13764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:00:50.310930 13772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:00:50.310974 13769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:00:50.311007 13770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:00:50.311034 13764 server_base.cc:1034] running on GCE node
I20260926 04:00:50.311349 13764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:00:50.311578 13764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:00:50.312707 13764 hybrid_clock.cc:648] HybridClock initialized: now 1790395250312677 us; error 39 us; skew 500 ppm
Sep 26 04:00:50 dist-test-slave-nhpw krb5kdc[13676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395250, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:00:50.315116 13764 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260926 04:00:50.315465 13764 webserver.cc:460] Webserver started at http://127.0.98.61:42631/ using document root <none> and password file <none>
I20260926 04:00:50.315656 13764 fs_manager.cc:359] Metadata directory not provided
I20260926 04:00:50.315701 13764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:00:50.315785 13764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:00:50.316812 13764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395095474643-392-0/minicluster-data/master-1/data/instance:
uuid: "0f33cb7c6c6947bf864f00be72fa31e3"
format_stamp: "Formatted at 2026-09-26 04:00:50 on dist-test-slave-nhpw"
server_key: "31dcca7652311ec2ae776e7d19cef14c"
server_key_iv: "cdd02ba4df66d73b720820d5b30b5b26"
server_key_version: "encryptionkey@0"
I20260926 04:00:50.316883 13757 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36457 } attempt: 2
I20260926 04:00:50.317176 13764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395095474643-392-0/minicluster-data/master-1/wal/instance:
uuid: "0f33cb7c6c6947bf864f00be72fa31e3"
format_stamp: "Formatted at 2026-09-26 04:00:50 on dist-test-slave-nhpw"
server_key: "31dcca7652311ec2ae776e7d19cef14c"
server_key_iv: "cdd02ba4df66d73b720820d5b30b5b26"
server_key_version: "encryptionkey@0"
I20260926 04:00:50.317309 13759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:50.317161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36457 (local address 127.0.0.1:57720)
0926 04:00:50.317224 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0926 04:00:50.317276 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 04:00:50.317638 13759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:50.317542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36457 (local address 127.0.0.1:57726)
0926 04:00:50.317592 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 04:00:50.317616 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36457: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260926 04:00:50.317734 13757 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36457: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36457: connect: Connection refused (error 111)
I20260926 04:00:50.318620 13764 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 04:00:50.319334 13779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:00:50.319499 13764 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 04:00:50.319592 13764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395095474643-392-0/minicluster-data/master-1/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395095474643-392-0/minicluster-data/master-1/wal
uuid: "0f33cb7c6c6947bf864f00be72fa31e3"
format_stamp: "Formatted at 2026-09-26 04:00:50 on dist-test-slave-nhpw"
server_key: "31dcca7652311ec2ae776e7d19cef14c"
server_key_iv: "cdd02ba4df66d73b720820d5b30b5b26"
server_key_version: "encryptionkey@0"
I20260926 04:00:50.319644 13764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395095474643-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395095474643-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395095474643-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:00:50.332113 13764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:00:50.334347 13764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:00:50.334450 13764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:00:50.338212 13764 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:36457
I20260926 04:00:50.338315 13831 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:36457 every 8 connection(s)
I20260926 04:00:50.338586 13764 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395095474643-392-0/minicluster-data/master-1/data/info.pb
I20260926 04:00:50.339470 13832 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:00:50.341061 13832 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37273 } has no permanent_uuid. Determining permanent_uuid...
I20260926 04:00:50.344013 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 13764
I20260926 04:00:50.344138 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395095474643-392-0/minicluster-data/master-1/wal/instance
I20260926 04:00:50.344331 392 external_mini_cluster.cc:1428] Setting key 1bf6e05c781b34e8845d445733e4db66
Sep 26 04:00:50 dist-test-slave-nhpw krb5kdc[13676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395250, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 04:00:50.350471 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:50.341776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55374 (local address 127.0.98.62:37273)
0926 04:00:50.341823 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 04:00:50.341825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:00:50.342017 (+ 192us) server_negotiation.cc:407] Connection header received
0926 04:00:50.342473 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:00:50.342474 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:00:50.342488 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:00:50.342552 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:00:50.343120 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:50.343606 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:50.344859 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:50.345210 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:50.345289 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:50.347551 (+ 2262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:00:50.347553 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:00:50.347553 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:00:50.347561 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:00:50.349063 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:00:50.349487 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:00:50.349488 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:00:50.349488 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:00:50.349552 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:00:50.349804 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:00:50.349805 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:00:50.349806 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:00:50.349920 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:00:50.349997 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 04:00:50.350275 (+ 278us) server_negotiation.cc:299] Negotiation successful
0926 04:00:50.350381 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 04:00:50.350621 13834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:50.341710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37273 (local address 127.0.0.1:55374)
0926 04:00:50.341871 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 04:00:50.341911 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 04:00:50.342350 (+ 439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:00:50.342622 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:00:50.342625 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:00:50.342859 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:00:50.343037 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:00:50.343040 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:00:50.343692 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:00:50.343694 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:00:50.344778 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:00:50.344780 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:00:50.345298 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:00:50.345298 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:00:50.345396 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:50.345914 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:00:50.345925 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:00:50.347452 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:00:50.349171 (+ 1719us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:00:50.349179 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:00:50.349181 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:00:50.349408 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:00:50.349620 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:00:50.349621 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:00:50.349621 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:00:50.349735 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:00:50.349988 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:00:50.349995 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:00:50.350136 (+ 141us) client_negotiation.cc:769] Sending connection context
0926 04:00:50.350254 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 04:00:50.350401 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":91,"threads_started":1}
I20260926 04:00:50.351099 13832 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36457 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 26 04:00:50 dist-test-slave-nhpw krb5kdc[13676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395250, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 04:00:50.359078 13834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:50.351385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36457 (local address 127.0.0.1:57732)
0926 04:00:50.351452 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0926 04:00:50.351463 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 04:00:50.351501 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:00:50.351794 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:00:50.351795 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:00:50.351942 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:00:50.352001 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:00:50.352003 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:00:50.352939 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:00:50.352940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:00:50.353568 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:00:50.353571 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:00:50.354021 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:00:50.354022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:00:50.354080 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:50.354509 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:00:50.354513 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:00:50.355856 (+ 1343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:00:50.357896 (+ 2040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:00:50.357896 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:00:50.357897 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:00:50.358112 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:00:50.358330 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:00:50.358331 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:00:50.358331 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:00:50.358401 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:00:50.358654 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:00:50.358655 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:00:50.358708 (+ 53us) client_negotiation.cc:769] Sending connection context
0926 04:00:50.358782 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 04:00:50.358892 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260926 04:00:50.359097 13836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:50.351451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57732 (local address 127.0.98.61:36457)
0926 04:00:50.351621 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 04:00:50.351626 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 04:00:50.351644 (+ 18us) server_negotiation.cc:407] Connection header received
0926 04:00:50.351670 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:00:50.351672 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:00:50.351682 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:00:50.351759 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:00:50.352078 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:50.352869 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:50.353641 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:50.353935 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:50.353994 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:50.355939 (+ 1945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:00:50.355947 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:00:50.355948 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:00:50.355954 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:00:50.357797 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:00:50.358194 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:00:50.358197 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:00:50.358197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:00:50.358258 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:00:50.358461 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:00:50.358462 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:00:50.358462 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:00:50.358587 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:00:50.358647 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 04:00:50.358817 (+ 170us) server_negotiation.cc:299] Negotiation successful
0926 04:00:50.358965 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":73,"server-negotiator.queue_time_us":144,"thread_start_us":95,"threads_started":1}
I20260926 04:00:50.359794 13832 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45123 } has no permanent_uuid. Determining permanent_uuid...
I20260926 04:00:50.360146 13834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:50.360007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45123 (local address 127.0.0.1:36726)
0926 04:00:50.360060 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 04:00:50.360113 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 04:00:50.360284 13780 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45123: connect: Connection refused (error 111)
I20260926 04:00:50.360710 13834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:50.360594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45123 (local address 127.0.0.1:36736)
0926 04:00:50.360622 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0926 04:00:50.360672 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260926 04:00:50.360858 13832 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45123: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45123: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395095474643-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395095474643-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260926 04:00:50.372408 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395095474643-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395095474643-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395095474643-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395095474643-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:45123
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395095474643-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37273,127.0.98.61:36457,127.0.98.60:45123
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 04:00:50.408870 13757 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36457 } attempt: 3
I20260926 04:00:50.410095 13832 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45123 } attempt: 1
I20260926 04:00:50.410470 13834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:50.410339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45123 (local address 127.0.0.1:36738)
0926 04:00:50.410389 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 04:00:50.410433 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 04:00:50.410789 13834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:50.410697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45123 (local address 127.0.0.1:36744)
0926 04:00:50.410744 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 04:00:50.410771 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 04:00:50.410907 13832 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45123: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45123: connect: Connection refused (error 111)
Sep 26 04:00:50 dist-test-slave-nhpw krb5kdc[13676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395250, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 04:00:50.416556 13836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:50.409459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57736 (local address 127.0.98.61:36457)
0926 04:00:50.409523 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 04:00:50.409525 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:00:50.409535 (+ 10us) server_negotiation.cc:407] Connection header received
0926 04:00:50.409641 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:00:50.409642 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:00:50.409659 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:00:50.409724 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:00:50.410048 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:50.410810 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:50.411545 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:50.411747 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:50.411824 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:50.413913 (+ 2089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:00:50.413915 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:00:50.413915 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:00:50.413922 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:00:50.415360 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:00:50.415769 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:00:50.415771 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:00:50.415771 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:00:50.415832 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:00:50.416037 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:00:50.416038 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:00:50.416039 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:00:50.416155 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:00:50.416229 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 04:00:50.416389 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 04:00:50.416466 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 04:00:50.416561 13759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:50.409368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36457 (local address 127.0.0.1:57736)
0926 04:00:50.409443 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0926 04:00:50.409464 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0926 04:00:50.409556 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:00:50.409732 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:00:50.409733 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:00:50.409901 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:00:50.409963 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:00:50.409965 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:00:50.410928 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:00:50.410929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:00:50.411465 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:00:50.411467 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:00:50.411823 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:00:50.411824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:00:50.411908 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:50.412414 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:00:50.412419 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:00:50.413812 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:00:50.415478 (+ 1666us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:00:50.415479 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:00:50.415479 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:00:50.415662 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:00:50.415903 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:00:50.415904 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:00:50.415904 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:00:50.415971 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:00:50.416234 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:00:50.416235 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:00:50.416279 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 04:00:50.416366 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 04:00:50.416466 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 04:00:50.417303 13757 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45123 } has no permanent_uuid. Determining permanent_uuid...
I20260926 04:00:50.417639 13759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:50.417542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45123 (local address 127.0.0.1:36752)
0926 04:00:50.417591 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 04:00:50.417620 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 04:00:50.417948 13759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:50.417859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45123 (local address 127.0.0.1:36766)
0926 04:00:50.417902 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0926 04:00:50.417925 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260926 04:00:50.418041 13757 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45123: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45123: connect: Connection refused (error 111)
I20260926 04:00:50.449198 13757 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45123 } attempt: 1
I20260926 04:00:50.449683 13759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:50.449539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45123 (local address 127.0.0.1:36782)
0926 04:00:50.449614 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0926 04:00:50.449654 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260926 04:00:50.450022 13759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:50.449926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45123 (local address 127.0.0.1:36796)
0926 04:00:50.449970 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 04:00:50.449997 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 04:00:50.450119 13757 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45123: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45123: connect: Connection refused (error 111)
W20260926 04:00:50.454501 13839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:00:50.454699 13839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:00:50.454720 13839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:00:50.455842 13839 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:00:50.455883 13839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:00:50.455897 13839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:00:50.455910 13839 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:00:50.455922 13839 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:00:50.457556 13839 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395095474643-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395095474643-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37273,127.0.98.61:36457,127.0.98.60:45123
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395095474643-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:45123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395095474643-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395095474643-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:00:50.457813 13839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:00:50.458040 13839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:00:50.463124 13844 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:00:50.463125 13847 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:00:50.463160 13845 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:00:50.463450 13839 server_base.cc:1034] running on GCE node
I20260926 04:00:50.463738 13839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:00:50.464022 13839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:00:50.465185 13839 hybrid_clock.cc:648] HybridClock initialized: now 1790395250465164 us; error 52 us; skew 500 ppm
Sep 26 04:00:50 dist-test-slave-nhpw krb5kdc[13676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395250, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:00:50.467749 13839 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260926 04:00:50.468216 13839 webserver.cc:460] Webserver started at http://127.0.98.60:35539/ using document root <none> and password file <none>
I20260926 04:00:50.468442 13839 fs_manager.cc:359] Metadata directory not provided
I20260926 04:00:50.468487 13839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:00:50.468681 13839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:00:50.469749 13839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395095474643-392-0/minicluster-data/master-2/data/instance:
uuid: "cabe24e5ee6b498fb7de34f52d54871e"
format_stamp: "Formatted at 2026-09-26 04:00:50 on dist-test-slave-nhpw"
server_key: "2d8bdb182817bef3956c433108dcde5d"
server_key_iv: "e4fddef3de17a2602523bbc534c1dbf6"
server_key_version: "encryptionkey@0"
I20260926 04:00:50.470108 13839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395095474643-392-0/minicluster-data/master-2/wal/instance:
uuid: "cabe24e5ee6b498fb7de34f52d54871e"
format_stamp: "Formatted at 2026-09-26 04:00:50 on dist-test-slave-nhpw"
server_key: "2d8bdb182817bef3956c433108dcde5d"
server_key_iv: "e4fddef3de17a2602523bbc534c1dbf6"
server_key_version: "encryptionkey@0"
I20260926 04:00:50.471442 13839 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 04:00:50.472220 13854 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:00:50.472443 13839 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:00:50.472551 13839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395095474643-392-0/minicluster-data/master-2/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395095474643-392-0/minicluster-data/master-2/wal
uuid: "cabe24e5ee6b498fb7de34f52d54871e"
format_stamp: "Formatted at 2026-09-26 04:00:50 on dist-test-slave-nhpw"
server_key: "2d8bdb182817bef3956c433108dcde5d"
server_key_iv: "e4fddef3de17a2602523bbc534c1dbf6"
server_key_version: "encryptionkey@0"
I20260926 04:00:50.472635 13839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395095474643-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395095474643-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395095474643-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:00:50.479058 13832 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45123 } attempt: 2
I20260926 04:00:50.479504 13834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:50.479374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45123 (local address 127.0.0.1:36804)
0926 04:00:50.479433 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0926 04:00:50.479465 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 04:00:50.479815 13834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:50.479725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45123 (local address 127.0.0.1:36816)
0926 04:00:50.479772 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 04:00:50.479795 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45123: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 04:00:50.479907 13832 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45123: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45123: connect: Connection refused (error 111)
I20260926 04:00:50.495676 13839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:00:50.498106 13839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:00:50.498282 13839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:00:50.502481 13839 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:45123
I20260926 04:00:50.502521 13906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:45123 every 8 connection(s)
I20260926 04:00:50.502895 13839 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395095474643-392-0/minicluster-data/master-2/data/info.pb
I20260926 04:00:50.503720 13907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:00:50.505219 13907 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37273 } has no permanent_uuid. Determining permanent_uuid...
I20260926 04:00:50.507800 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 13839
I20260926 04:00:50.507900 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790395095474643-392-0/minicluster-data/master-2/wal/instance
I20260926 04:00:50.508064 392 external_mini_cluster.cc:1428] Setting key 07a1f132023d94d9bf46691b22f6f477
Sep 26 04:00:50 dist-test-slave-nhpw krb5kdc[13676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395250, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 26 04:00:50 dist-test-slave-nhpw krb5kdc[13676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 04:00:50.514981 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:50.505873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55386 (local address 127.0.98.62:37273)
0926 04:00:50.505979 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0926 04:00:50.505981 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:00:50.506097 (+ 116us) server_negotiation.cc:407] Connection header received
0926 04:00:50.506537 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:00:50.506538 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:00:50.506547 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:00:50.506599 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:00:50.507147 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:50.507613 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:50.509192 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:50.509382 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:50.509440 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:50.511831 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:00:50.511833 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:00:50.511833 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:00:50.511841 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:00:50.513480 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:00:50.513989 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:00:50.513991 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:00:50.513991 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:00:50.514057 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:00:50.514331 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:00:50.514332 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:00:50.514333 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:00:50.514452 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:00:50.514528 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 04:00:50.514801 (+ 273us) server_negotiation.cc:299] Negotiation successful
0926 04:00:50.514885 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 04:00:50.515180 13909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:50.505815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37273 (local address 127.0.0.1:55386)
0926 04:00:50.505974 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 04:00:50.506016 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 04:00:50.506452 (+ 436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:00:50.506652 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:00:50.506654 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:00:50.506889 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:00:50.507073 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:00:50.507076 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:00:50.507707 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:00:50.507708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:00:50.509097 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:00:50.509100 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:00:50.509468 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:00:50.509469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:00:50.509617 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:50.510163 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:00:50.510172 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:00:50.511733 (+ 1561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:00:50.513588 (+ 1855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:00:50.513596 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:00:50.513598 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:00:50.513909 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:00:50.514130 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:00:50.514131 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:00:50.514131 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:00:50.514260 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:00:50.514519 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:00:50.514526 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:00:50.514669 (+ 143us) client_negotiation.cc:769] Sending connection context
0926 04:00:50.514782 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 04:00:50.514935 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":64,"threads_started":1}
I20260926 04:00:50.515789 13907 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36457 } has no permanent_uuid. Determining permanent_uuid...
I20260926 04:00:50.517613 13911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:50.508438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36826 (local address 127.0.98.60:45123)
0926 04:00:50.508567 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 04:00:50.508572 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 04:00:50.508617 (+ 45us) server_negotiation.cc:407] Connection header received
0926 04:00:50.508733 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:00:50.508735 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:00:50.508752 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:00:50.508807 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:00:50.509524 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:50.510367 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:50.511244 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:50.511664 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:50.511749 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:50.514136 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:00:50.514144 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:00:50.514144 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:00:50.514151 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:00:50.515989 (+ 1838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:00:50.516407 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:00:50.516409 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:00:50.516410 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:00:50.516471 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:00:50.516785 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:00:50.516786 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:00:50.516786 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:00:50.517033 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:00:50.517252 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0926 04:00:50.517333 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 04:00:50.517427 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":75,"threads_started":1}
Sep 26 04:00:50 dist-test-slave-nhpw krb5kdc[13676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395250, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 04:00:50.523468 13836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:50.516159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57740 (local address 127.0.98.61:36457)
0926 04:00:50.516211 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 04:00:50.516215 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 04:00:50.516260 (+ 45us) server_negotiation.cc:407] Connection header received
0926 04:00:50.516336 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:00:50.516337 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:00:50.516346 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:00:50.516417 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:00:50.516781 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:50.517604 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:50.518379 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:50.518666 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:50.518767 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:50.520715 (+ 1948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:00:50.520716 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:00:50.520717 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:00:50.520739 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:00:50.522352 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:00:50.522690 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:00:50.522691 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:00:50.522691 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:00:50.522752 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:00:50.522936 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:00:50.522937 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:00:50.522937 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:00:50.523055 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:00:50.523108 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 04:00:50.523306 (+ 198us) server_negotiation.cc:299] Negotiation successful
0926 04:00:50.523395 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 04:00:50.523491 13909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:50.516099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36457 (local address 127.0.0.1:57740)
0926 04:00:50.516215 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 04:00:50.516226 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 04:00:50.516275 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:00:50.516421 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:00:50.516423 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:00:50.516583 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:00:50.516615 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:00:50.516616 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:00:50.517758 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:00:50.517759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:00:50.518293 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:00:50.518294 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:00:50.518764 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:00:50.518765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:00:50.518824 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:50.519211 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:00:50.519216 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:00:50.520594 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:00:50.522444 (+ 1850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:00:50.522445 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:00:50.522445 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:00:50.522602 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:00:50.522819 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:00:50.522819 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:00:50.522820 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:00:50.522870 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:00:50.523143 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:00:50.523144 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:00:50.523196 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 04:00:50.523275 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 04:00:50.523392 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98}
Sep 26 04:00:50 dist-test-slave-nhpw krb5kdc[13676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 04:00:50.523904 13907 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45123 } has no permanent_uuid. Determining permanent_uuid...
I20260926 04:00:50.525280 13757 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45123 } attempt: 2
I20260926 04:00:50.527222 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:50.518666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55390 (local address 127.0.98.62:37273)
0926 04:00:50.518706 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 04:00:50.518707 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 04:00:50.518866 (+ 159us) server_negotiation.cc:407] Connection header received
0926 04:00:50.518954 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:00:50.518955 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:00:50.518967 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:00:50.519026 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:00:50.519594 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:50.520303 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:50.521266 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:50.521483 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:50.521556 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:50.524067 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:00:50.524069 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:00:50.524069 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:00:50.524077 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:00:50.526059 (+ 1982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:00:50.526394 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:00:50.526395 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:00:50.526395 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:00:50.526450 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:00:50.526676 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:00:50.526677 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:00:50.526677 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:00:50.526789 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:00:50.527001 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0926 04:00:50.527039 (+ 38us) server_negotiation.cc:299] Negotiation successful
0926 04:00:50.527109 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
Sep 26 04:00:50 dist-test-slave-nhpw krb5kdc[13676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395250, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 26 04:00:50 dist-test-slave-nhpw krb5kdc[13676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395250, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 26 04:00:50 dist-test-slave-nhpw krb5kdc[13676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 04:00:50.532617 13909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:50.524169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45123 (local address 127.0.0.1:36832)
0926 04:00:50.524356 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 04:00:50.524365 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 04:00:50.524416 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:00:50.524534 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:00:50.524535 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:00:50.525026 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:00:50.525065 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:00:50.525067 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:00:50.525847 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:00:50.525848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:00:50.526592 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:00:50.526594 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:00:50.527168 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:00:50.527168 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:00:50.527222 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:50.527626 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:00:50.527632 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:00:50.529024 (+ 1392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:00:50.531539 (+ 2515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:00:50.531539 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:00:50.531540 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:00:50.531743 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:00:50.531939 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:00:50.531940 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:00:50.531941 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:00:50.531998 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:00:50.532266 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:00:50.532267 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:00:50.532313 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 04:00:50.532401 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 04:00:50.532514 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173}
I20260926 04:00:50.532999 13911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:50.524251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36832 (local address 127.0.98.60:45123)
0926 04:00:50.524294 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 04:00:50.524296 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:00:50.524440 (+ 144us) server_negotiation.cc:407] Connection header received
0926 04:00:50.524463 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:00:50.524464 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:00:50.524474 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:00:50.524529 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:00:50.525135 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:50.525762 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:50.526667 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:50.526957 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:50.527110 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:50.529097 (+ 1987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:00:50.529098 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:00:50.529099 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:00:50.529105 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:00:50.531453 (+ 2348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:00:50.531811 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:00:50.531812 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:00:50.531813 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:00:50.531863 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:00:50.532065 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:00:50.532066 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:00:50.532067 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:00:50.532186 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:00:50.532254 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 04:00:50.532442 (+ 188us) server_negotiation.cc:299] Negotiation successful
0926 04:00:50.532604 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":22}
I20260926 04:00:50.534126 13759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:50.525517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45123 (local address 127.0.0.1:36844)
0926 04:00:50.525554 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0926 04:00:50.525564 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 04:00:50.525602 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:00:50.525905 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:00:50.525906 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:00:50.526114 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:00:50.526150 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:00:50.526152 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:00:50.527131 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:00:50.527132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:00:50.528221 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:00:50.528225 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:00:50.528796 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:00:50.528797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:00:50.528855 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:50.529264 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:00:50.529269 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:00:50.530641 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:00:50.532821 (+ 2180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:00:50.532822 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:00:50.532822 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:00:50.533025 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:00:50.533466 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:00:50.533467 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:00:50.533467 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:00:50.533524 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:00:50.533794 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:00:50.533795 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:00:50.533838 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 04:00:50.533934 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 04:00:50.534025 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260926 04:00:50.534209 13912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:50.525562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36844 (local address 127.0.98.60:45123)
0926 04:00:50.525752 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 04:00:50.525754 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:00:50.525762 (+ 8us) server_negotiation.cc:407] Connection header received
0926 04:00:50.525781 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:00:50.525782 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:00:50.525792 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:00:50.525892 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:00:50.526251 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:50.526957 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:50.528319 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:50.528697 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:50.528992 (+ 295us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:50.530762 (+ 1770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:00:50.530763 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:00:50.530764 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:00:50.530770 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:00:50.532724 (+ 1954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:00:50.533357 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:00:50.533357 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:00:50.533358 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:00:50.533413 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:00:50.533596 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:00:50.533597 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:00:50.533597 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:00:50.533722 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:00:50.533802 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 04:00:50.533988 (+ 186us) server_negotiation.cc:299] Negotiation successful
0926 04:00:50.534066 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":109,"threads_started":1}
I20260926 04:00:50.534734 13907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cabe24e5ee6b498fb7de34f52d54871e: Bootstrap starting.
I20260926 04:00:50.535545 13907 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cabe24e5ee6b498fb7de34f52d54871e: Neither blocks nor log segments found. Creating new log.
I20260926 04:00:50.535835 13907 log.cc:824] T 00000000000000000000000000000000 P cabe24e5ee6b498fb7de34f52d54871e: Log is configured to *not* fsync() on all Append() calls
I20260926 04:00:50.536034 13836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:50.528093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57742 (local address 127.0.98.61:36457)
0926 04:00:50.528131 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 04:00:50.528133 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:00:50.528141 (+ 8us) server_negotiation.cc:407] Connection header received
0926 04:00:50.528194 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:00:50.528195 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:00:50.528203 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:00:50.528258 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:00:50.528756 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:50.529461 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:50.530360 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:50.530607 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:50.530640 (+ 33us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:50.533184 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:00:50.533185 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:00:50.533186 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:00:50.533193 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:00:50.534848 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:00:50.535240 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:00:50.535241 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:00:50.535241 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:00:50.535292 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:00:50.535495 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:00:50.535496 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:00:50.535496 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:00:50.535611 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:00:50.535822 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0926 04:00:50.535859 (+ 37us) server_negotiation.cc:299] Negotiation successful
0926 04:00:50.535936 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 04:00:50.536715 13907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cabe24e5ee6b498fb7de34f52d54871e: No bootstrap required, opened a new log
I20260926 04:00:50.537530 13757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69a8e46547974380b7c57d002f309c19: Bootstrap starting.
I20260926 04:00:50.537694 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 04:00:50.538228 13757 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69a8e46547974380b7c57d002f309c19: Neither blocks nor log segments found. Creating new log.
I20260926 04:00:50.538511 13757 log.cc:824] T 00000000000000000000000000000000 P 69a8e46547974380b7c57d002f309c19: Log is configured to *not* fsync() on all Append() calls
I20260926 04:00:50.538861 13907 raft_consensus.cc:348] T 00000000000000000000000000000000 P cabe24e5ee6b498fb7de34f52d54871e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69a8e46547974380b7c57d002f309c19" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37273 } } peers { permanent_uuid: "0f33cb7c6c6947bf864f00be72fa31e3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36457 } } peers { permanent_uuid: "cabe24e5ee6b498fb7de34f52d54871e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45123 } }
I20260926 04:00:50.539016 13907 raft_consensus.cc:374] T 00000000000000000000000000000000 P cabe24e5ee6b498fb7de34f52d54871e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:00:50.539054 13907 raft_consensus.cc:729] T 00000000000000000000000000000000 P cabe24e5ee6b498fb7de34f52d54871e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cabe24e5ee6b498fb7de34f52d54871e, State: Initialized, Role: FOLLOWER
I20260926 04:00:50.539285 13907 consensus_queue.cc:260] T 00000000000000000000000000000000 P cabe24e5ee6b498fb7de34f52d54871e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69a8e46547974380b7c57d002f309c19" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37273 } } peers { permanent_uuid: "0f33cb7c6c6947bf864f00be72fa31e3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36457 } } peers { permanent_uuid: "cabe24e5ee6b498fb7de34f52d54871e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45123 } }
I20260926 04:00:50.539738 13907 sys_catalog.cc:564] T 00000000000000000000000000000000 P cabe24e5ee6b498fb7de34f52d54871e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:00:50.539978 13757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69a8e46547974380b7c57d002f309c19: No bootstrap required, opened a new log
I20260926 04:00:50.539983 13922 sys_catalog.cc:455] T 00000000000000000000000000000000 P cabe24e5ee6b498fb7de34f52d54871e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69a8e46547974380b7c57d002f309c19" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37273 } } peers { permanent_uuid: "0f33cb7c6c6947bf864f00be72fa31e3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36457 } } peers { permanent_uuid: "cabe24e5ee6b498fb7de34f52d54871e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45123 } } }
I20260926 04:00:50.540051 13922 sys_catalog.cc:458] T 00000000000000000000000000000000 P cabe24e5ee6b498fb7de34f52d54871e [sys.catalog]: This master's current role is: FOLLOWER
W20260926 04:00:50.541661 13933 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cabe24e5ee6b498fb7de34f52d54871e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260926 04:00:50.542266 13757 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69a8e46547974380b7c57d002f309c19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69a8e46547974380b7c57d002f309c19" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37273 } } peers { permanent_uuid: "0f33cb7c6c6947bf864f00be72fa31e3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36457 } } peers { permanent_uuid: "cabe24e5ee6b498fb7de34f52d54871e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45123 } }
W20260926 04:00:50.542351 13933 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 04:00:50.542387 13757 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69a8e46547974380b7c57d002f309c19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:00:50.542416 13757 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69a8e46547974380b7c57d002f309c19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69a8e46547974380b7c57d002f309c19, State: Initialized, Role: FOLLOWER
I20260926 04:00:50.542516 13757 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69a8e46547974380b7c57d002f309c19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69a8e46547974380b7c57d002f309c19" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37273 } } peers { permanent_uuid: "0f33cb7c6c6947bf864f00be72fa31e3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36457 } } peers { permanent_uuid: "cabe24e5ee6b498fb7de34f52d54871e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45123 } }
I20260926 04:00:50.542804 13757 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69a8e46547974380b7c57d002f309c19 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:00:50.542815 13935 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69a8e46547974380b7c57d002f309c19 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69a8e46547974380b7c57d002f309c19" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37273 } } peers { permanent_uuid: "0f33cb7c6c6947bf864f00be72fa31e3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36457 } } peers { permanent_uuid: "cabe24e5ee6b498fb7de34f52d54871e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45123 } } }
I20260926 04:00:50.544284 13935 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69a8e46547974380b7c57d002f309c19 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 04:00:50.547148 13760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:50.538996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55394 (local address 127.0.98.62:37273)
0926 04:00:50.539033 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 04:00:50.539035 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:00:50.539253 (+ 218us) server_negotiation.cc:407] Connection header received
0926 04:00:50.539277 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:00:50.539277 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:00:50.539287 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:00:50.539346 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:00:50.540170 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:50.540958 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:50.541957 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:50.542260 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:50.542323 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:50.543877 (+ 1554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:00:50.543878 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:00:50.543879 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:00:50.543885 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:00:50.545509 (+ 1624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:00:50.545989 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:00:50.545991 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:00:50.545991 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:00:50.546044 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:00:50.546258 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:00:50.546259 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:00:50.546260 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:00:50.546392 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:00:50.546486 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 04:00:50.546946 (+ 460us) server_negotiation.cc:299] Negotiation successful
0926 04:00:50.547027 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 04:00:50.545590 13689 master_runner.cc:186] Error getting master registration for 127.0.98.61:36457: OK, instance_id { permanent_uuid: "0f33cb7c6c6947bf864f00be72fa31e3" instance_seqno: 1790395250337154 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 04:00:50.548372 13689 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 04:00:50.549361 13946 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 69a8e46547974380b7c57d002f309c19: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 04:00:50.549420 13946 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 04:00:50.550561 13836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:50.539195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57754 (local address 127.0.98.61:36457)
0926 04:00:50.539235 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 04:00:50.539236 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 04:00:50.539471 (+ 235us) server_negotiation.cc:407] Connection header received
0926 04:00:50.539577 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:00:50.539578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:00:50.539589 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:00:50.539619 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:00:50.543247 (+ 3628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:50.543989 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:50.545789 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:50.546090 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:50.546132 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:50.547638 (+ 1506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:00:50.547639 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:00:50.547640 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:00:50.547647 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:00:50.549309 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:00:50.549704 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:00:50.549705 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:00:50.549705 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:00:50.549763 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:00:50.549959 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:00:50.549959 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:00:50.549960 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:00:50.550088 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:00:50.550165 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 04:00:50.550385 (+ 220us) server_negotiation.cc:299] Negotiation successful
0926 04:00:50.550470 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 04:00:50.549136 13839 master_runner.cc:186] Error getting master registration for 127.0.98.61:36457: OK, instance_id { permanent_uuid: "0f33cb7c6c6947bf864f00be72fa31e3" instance_seqno: 1790395250337154 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 04:00:50.552095 13839 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 04:00:50.552842 13912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:50.539436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36858 (local address 127.0.98.60:45123)
0926 04:00:50.539471 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 04:00:50.539473 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:00:50.540872 (+ 1399us) server_negotiation.cc:407] Connection header received
0926 04:00:50.540966 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:00:50.540967 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:00:50.540976 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:00:50.541007 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:00:50.541736 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:50.542755 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:50.543923 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:50.544219 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:50.544266 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:50.549990 (+ 5724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:00:50.549991 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:00:50.549992 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:00:50.549999 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:00:50.551592 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:00:50.551987 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:00:50.551988 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:00:50.551988 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:00:50.552048 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:00:50.552268 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:00:50.552269 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:00:50.552269 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:00:50.552384 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:00:50.552504 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 04:00:50.552609 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 04:00:50.552731 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 04:00:50.571050 13832 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45123 } attempt: 3
Sep 26 04:00:50 dist-test-slave-nhpw krb5kdc[13676](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395250, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 04:00:50.578593 13834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:50.571437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45123 (local address 127.0.0.1:36862)
0926 04:00:50.571494 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 04:00:50.571509 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 04:00:50.571556 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:00:50.571717 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:00:50.571717 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:00:50.571973 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:00:50.572028 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:00:50.572031 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:00:50.572972 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:00:50.572973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:00:50.573498 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:00:50.573500 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:00:50.573882 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:00:50.573884 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:00:50.573942 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:50.574419 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:00:50.574424 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:00:50.575763 (+ 1339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:00:50.577514 (+ 1751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:00:50.577516 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:00:50.577516 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:00:50.577705 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:00:50.577922 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:00:50.577923 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:00:50.577924 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:00:50.577982 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:00:50.578255 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:00:50.578257 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:00:50.578301 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 04:00:50.578389 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 04:00:50.578489 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 04:00:50.578593 13912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:50.571567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36862 (local address 127.0.98.60:45123)
0926 04:00:50.571608 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 04:00:50.571610 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:00:50.571618 (+ 8us) server_negotiation.cc:407] Connection header received
0926 04:00:50.571641 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:00:50.571641 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:00:50.571650 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:00:50.571706 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:00:50.572113 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:50.572892 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:50.573585 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:50.573798 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:50.573872 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:50.575852 (+ 1980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:00:50.575854 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:00:50.575854 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:00:50.575862 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:00:50.577404 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:00:50.577794 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:00:50.577795 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:00:50.577796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:00:50.577850 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:00:50.578061 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:00:50.578062 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:00:50.578062 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:00:50.578183 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:00:50.578254 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 04:00:50.578413 (+ 159us) server_negotiation.cc:299] Negotiation successful
0926 04:00:50.578489 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 04:00:50.580224 13832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f33cb7c6c6947bf864f00be72fa31e3: Bootstrap starting.
I20260926 04:00:50.581019 13832 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f33cb7c6c6947bf864f00be72fa31e3: Neither blocks nor log segments found. Creating new log.
I20260926 04:00:50.581326 13832 log.cc:824] T 00000000000000000000000000000000 P 0f33cb7c6c6947bf864f00be72fa31e3: Log is configured to *not* fsync() on all Append() calls
I20260926 04:00:50.582386 13832 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f33cb7c6c6947bf864f00be72fa31e3: No bootstrap required, opened a new log
I20260926 04:00:50.583792 13832 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f33cb7c6c6947bf864f00be72fa31e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69a8e46547974380b7c57d002f309c19" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37273 } } peers { permanent_uuid: "0f33cb7c6c6947bf864f00be72fa31e3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36457 } } peers { permanent_uuid: "cabe24e5ee6b498fb7de34f52d54871e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45123 } }
I20260926 04:00:50.583889 13832 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f33cb7c6c6947bf864f00be72fa31e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:00:50.583910 13832 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f33cb7c6c6947bf864f00be72fa31e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f33cb7c6c6947bf864f00be72fa31e3, State: Initialized, Role: FOLLOWER
I20260926 04:00:50.584031 13832 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f33cb7c6c6947bf864f00be72fa31e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69a8e46547974380b7c57d002f309c19" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37273 } } peers { permanent_uuid: "0f33cb7c6c6947bf864f00be72fa31e3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36457 } } peers { permanent_uuid: "cabe24e5ee6b498fb7de34f52d54871e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45123 } }
I20260926 04:00:50.584285 13949 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f33cb7c6c6947bf864f00be72fa31e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69a8e46547974380b7c57d002f309c19" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37273 } } peers { permanent_uuid: "0f33cb7c6c6947bf864f00be72fa31e3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36457 } } peers { permanent_uuid: "cabe24e5ee6b498fb7de34f52d54871e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45123 } } }
I20260926 04:00:50.584349 13949 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f33cb7c6c6947bf864f00be72fa31e3 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 04:00:50.584352 13832 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f33cb7c6c6947bf864f00be72fa31e3 [sys.catalog]: configured and running, proceeding with master startup.
W20260926 04:00:50.584954 13785 tablet.cc:2369] T 00000000000000000000000000000000 P 0f33cb7c6c6947bf864f00be72fa31e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 04:00:50.586270 13960 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0f33cb7c6c6947bf864f00be72fa31e3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 04:00:50.586391 13960 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 04:00:50.588223 13764 master_runner.cc:263] No leader master found from master 0f33cb7c6c6947bf864f00be72fa31e3
I20260926 04:00:50.588266 13764 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 04:00:50.680883 13710 tablet.cc:2369] T 00000000000000000000000000000000 P 69a8e46547974380b7c57d002f309c19: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 04:00:50.748968 13860 tablet.cc:2369] T 00000000000000000000000000000000 P cabe24e5ee6b498fb7de34f52d54871e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 04:00:50.957121 13949 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f33cb7c6c6947bf864f00be72fa31e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 04:00:50.957280 13949 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f33cb7c6c6947bf864f00be72fa31e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69a8e46547974380b7c57d002f309c19" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37273 } } peers { permanent_uuid: "0f33cb7c6c6947bf864f00be72fa31e3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36457 } } peers { permanent_uuid: "cabe24e5ee6b498fb7de34f52d54871e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45123 } }
I20260926 04:00:50.957583 13949 leader_election.cc:288] T 00000000000000000000000000000000 P 0f33cb7c6c6947bf864f00be72fa31e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 69a8e46547974380b7c57d002f309c19 (127.0.98.62:37273), cabe24e5ee6b498fb7de34f52d54871e (127.0.98.60:45123)
I20260926 04:00:50.957906 13731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0f33cb7c6c6947bf864f00be72fa31e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69a8e46547974380b7c57d002f309c19" is_pre_election: true
I20260926 04:00:50.957906 13881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0f33cb7c6c6947bf864f00be72fa31e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cabe24e5ee6b498fb7de34f52d54871e" is_pre_election: true
I20260926 04:00:50.958086 13881 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cabe24e5ee6b498fb7de34f52d54871e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f33cb7c6c6947bf864f00be72fa31e3 in term 0.
I20260926 04:00:50.958120 13731 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 69a8e46547974380b7c57d002f309c19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f33cb7c6c6947bf864f00be72fa31e3 in term 0.
I20260926 04:00:50.958302 13780 leader_election.cc:302] T 00000000000000000000000000000000 P 0f33cb7c6c6947bf864f00be72fa31e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f33cb7c6c6947bf864f00be72fa31e3, cabe24e5ee6b498fb7de34f52d54871e; no voters:
I20260926 04:00:50.958460 13949 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f33cb7c6c6947bf864f00be72fa31e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 04:00:50.958521 13949 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f33cb7c6c6947bf864f00be72fa31e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 04:00:50.958546 13949 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f33cb7c6c6947bf864f00be72fa31e3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:00:50.959336 13949 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f33cb7c6c6947bf864f00be72fa31e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69a8e46547974380b7c57d002f309c19" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37273 } } peers { permanent_uuid: "0f33cb7c6c6947bf864f00be72fa31e3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36457 } } peers { permanent_uuid: "cabe24e5ee6b498fb7de34f52d54871e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45123 } }
I20260926 04:00:50.959478 13949 leader_election.cc:288] T 00000000000000000000000000000000 P 0f33cb7c6c6947bf864f00be72fa31e3 [CANDIDATE]: Term 1 election: Requested vote from peers 69a8e46547974380b7c57d002f309c19 (127.0.98.62:37273), cabe24e5ee6b498fb7de34f52d54871e (127.0.98.60:45123)
I20260926 04:00:50.959663 13881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0f33cb7c6c6947bf864f00be72fa31e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cabe24e5ee6b498fb7de34f52d54871e"
I20260926 04:00:50.959748 13731 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0f33cb7c6c6947bf864f00be72fa31e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69a8e46547974380b7c57d002f309c19"
I20260926 04:00:50.959785 13881 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cabe24e5ee6b498fb7de34f52d54871e [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:00:50.959851 13731 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69a8e46547974380b7c57d002f309c19 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:00:50.960492 13881 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cabe24e5ee6b498fb7de34f52d54871e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f33cb7c6c6947bf864f00be72fa31e3 in term 1.
I20260926 04:00:50.960675 13731 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 69a8e46547974380b7c57d002f309c19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f33cb7c6c6947bf864f00be72fa31e3 in term 1.
I20260926 04:00:50.960702 13780 leader_election.cc:302] T 00000000000000000000000000000000 P 0f33cb7c6c6947bf864f00be72fa31e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f33cb7c6c6947bf864f00be72fa31e3, cabe24e5ee6b498fb7de34f52d54871e; no voters:
I20260926 04:00:50.960785 13949 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f33cb7c6c6947bf864f00be72fa31e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:00:50.960955 13949 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f33cb7c6c6947bf864f00be72fa31e3 [term 1 LEADER]: Becoming Leader. State: Replica: 0f33cb7c6c6947bf864f00be72fa31e3, State: Running, Role: LEADER
I20260926 04:00:50.961057 13949 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f33cb7c6c6947bf864f00be72fa31e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69a8e46547974380b7c57d002f309c19" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37273 } } peers { permanent_uuid: "0f33cb7c6c6947bf864f00be72fa31e3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36457 } } peers { permanent_uuid: "cabe24e5ee6b498fb7de34f52d54871e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45123 } }
I20260926 04:00:50.961485 13962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f33cb7c6c6947bf864f00be72fa31e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f33cb7c6c6947bf864f00be72fa31e3. Latest consensus state: current_term: 1 leader_uuid: "0f33cb7c6c6947bf864f00be72fa31e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69a8e46547974380b7c57d002f309c19" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37273 } } peers { permanent_uuid: "0f33cb7c6c6947bf864f00be72fa31e3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36457 } } peers { permanent_uuid: "cabe24e5ee6b498fb7de34f52d54871e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45123 } } }
I20260926 04:00:50.961558 13962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f33cb7c6c6947bf864f00be72fa31e3 [sys.catalog]: This master's current role is: LEADER
I20260926 04:00:50.961752 13964 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:00:50.961891 13964 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:00:50.963227 13731 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 69a8e46547974380b7c57d002f309c19 [term 1 FOLLOWER]: Refusing update from remote peer 0f33cb7c6c6947bf864f00be72fa31e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:00:50.963232 13881 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cabe24e5ee6b498fb7de34f52d54871e [term 1 FOLLOWER]: Refusing update from remote peer 0f33cb7c6c6947bf864f00be72fa31e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:00:50.963514 13949 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0f33cb7c6c6947bf864f00be72fa31e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cabe24e5ee6b498fb7de34f52d54871e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45123 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:00:50.963631 13962 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0f33cb7c6c6947bf864f00be72fa31e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69a8e46547974380b7c57d002f309c19" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:00:50.964777 13922 sys_catalog.cc:455] T 00000000000000000000000000000000 P cabe24e5ee6b498fb7de34f52d54871e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f33cb7c6c6947bf864f00be72fa31e3. Latest consensus state: current_term: 1 leader_uuid: "0f33cb7c6c6947bf864f00be72fa31e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69a8e46547974380b7c57d002f309c19" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37273 } } peers { permanent_uuid: "0f33cb7c6c6947bf864f00be72fa31e3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36457 } } peers { permanent_uuid: "cabe24e5ee6b498fb7de34f52d54871e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45123 } } }
I20260926 04:00:50.964793 13935 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69a8e46547974380b7c57d002f309c19 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f33cb7c6c6947bf864f00be72fa31e3. Latest consensus state: current_term: 1 leader_uuid: "0f33cb7c6c6947bf864f00be72fa31e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69a8e46547974380b7c57d002f309c19" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37273 } } peers { permanent_uuid: "0f33cb7c6c6947bf864f00be72fa31e3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36457 } } peers { permanent_uuid: "cabe24e5ee6b498fb7de34f52d54871e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45123 } } }
I20260926 04:00:50.964871 13935 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69a8e46547974380b7c57d002f309c19 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 04:00:50.964865 13922 sys_catalog.cc:458] T 00000000000000000000000000000000 P cabe24e5ee6b498fb7de34f52d54871e [sys.catalog]: This master's current role is: FOLLOWER
I20260926 04:00:50.965590 13962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f33cb7c6c6947bf864f00be72fa31e3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0f33cb7c6c6947bf864f00be72fa31e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69a8e46547974380b7c57d002f309c19" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37273 } } peers { permanent_uuid: "0f33cb7c6c6947bf864f00be72fa31e3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36457 } } peers { permanent_uuid: "cabe24e5ee6b498fb7de34f52d54871e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45123 } } }
I20260926 04:00:50.965670 13962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f33cb7c6c6947bf864f00be72fa31e3 [sys.catalog]: This master's current role is: LEADER
I20260926 04:00:50.965725 13962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f33cb7c6c6947bf864f00be72fa31e3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0f33cb7c6c6947bf864f00be72fa31e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69a8e46547974380b7c57d002f309c19" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37273 } } peers { permanent_uuid: "0f33cb7c6c6947bf864f00be72fa31e3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36457 } } peers { permanent_uuid: "cabe24e5ee6b498fb7de34f52d54871e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45123 } } }
I20260926 04:00:50.965786 13962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f33cb7c6c6947bf864f00be72fa31e3 [sys.catalog]: This master's current role is: LEADER
I20260926 04:00:50.965816 13935 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69a8e46547974380b7c57d002f309c19 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0f33cb7c6c6947bf864f00be72fa31e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69a8e46547974380b7c57d002f309c19" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37273 } } peers { permanent_uuid: "0f33cb7c6c6947bf864f00be72fa31e3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36457 } } peers { permanent_uuid: "cabe24e5ee6b498fb7de34f52d54871e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45123 } } }
I20260926 04:00:50.965886 13935 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69a8e46547974380b7c57d002f309c19 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 04:00:50.965993 13964 catalog_manager.cc:1347] Generated new cluster ID: 90ca973d4e1546d292861d4ac2d9a259
I20260926 04:00:50.965988 13967 mvcc.cc:205] Tried to move back new op lower bound from 7333458947942883328 to 7333458947937222656. Current Snapshot: MvccSnapshot[applied={T|T < 7333458947942883328}]
I20260926 04:00:50.966035 13964 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 04:00:50.966012 13922 sys_catalog.cc:455] T 00000000000000000000000000000000 P cabe24e5ee6b498fb7de34f52d54871e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0f33cb7c6c6947bf864f00be72fa31e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69a8e46547974380b7c57d002f309c19" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37273 } } peers { permanent_uuid: "0f33cb7c6c6947bf864f00be72fa31e3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36457 } } peers { permanent_uuid: "cabe24e5ee6b498fb7de34f52d54871e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45123 } } }
I20260926 04:00:50.966080 13922 sys_catalog.cc:458] T 00000000000000000000000000000000 P cabe24e5ee6b498fb7de34f52d54871e [sys.catalog]: This master's current role is: FOLLOWER
I20260926 04:00:50.966339 13966 mvcc.cc:205] Tried to move back new op lower bound from 7333458947942883328 to 7333458947937222656. Current Snapshot: MvccSnapshot[applied={T|T < 7333458947942883328}]
I20260926 04:00:50.981101 13964 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:00:50.981675 13964 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:00:50.984332 13964 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f33cb7c6c6947bf864f00be72fa31e3: Generated new TSK 0
I20260926 04:00:50.984578 13964 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 04:00:51.542842 13933 catalog_manager.cc:1259] Loaded cluster ID: 90ca973d4e1546d292861d4ac2d9a259
I20260926 04:00:51.542927 13933 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cabe24e5ee6b498fb7de34f52d54871e: loading cluster ID for follower catalog manager: success
I20260926 04:00:51.543787 13933 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cabe24e5ee6b498fb7de34f52d54871e: acquiring CA information for follower catalog manager: success
I20260926 04:00:51.544104 13933 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cabe24e5ee6b498fb7de34f52d54871e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 04:00:51.549808 13946 catalog_manager.cc:1259] Loaded cluster ID: 90ca973d4e1546d292861d4ac2d9a259
I20260926 04:00:51.549863 13946 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 69a8e46547974380b7c57d002f309c19: loading cluster ID for follower catalog manager: success
I20260926 04:00:51.550608 13946 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 69a8e46547974380b7c57d002f309c19: acquiring CA information for follower catalog manager: success
I20260926 04:00:51.550850 13946 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 69a8e46547974380b7c57d002f309c19: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 04:00:52.045198 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 13689
I20260926 04:00:52.050825 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 13764
I20260926 04:00:52.055862 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 13839
2026-09-26T04:00:52Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3135 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790395095474643-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 04:00:52 dist-test-slave-nhpw krb5kdc[13979](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 04:00:52 dist-test-slave-nhpw krb5kdc[13979](info): set up 2 sockets
Sep 26 04:00:52 dist-test-slave-nhpw krb5kdc[13979](info): commencing operation
krb5kdc: starting...
W20260926 04:00:53.096776 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790395095474643-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790395095474643-392-0/krb5kdc/test-user.keytab.
Sep 26 04:00:53 dist-test-slave-nhpw krb5kdc[13979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T04:00:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:00:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 04:00:53.264065 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790395095474643-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790395095474643-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790395095474643-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34813
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34813
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:00:53.345289 13992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:00:53.345530 13992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:00:53.345568 13992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:00:53.346716 13992 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:00:53.346789 13992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:00:53.346817 13992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:00:53.346858 13992 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:00:53.346887 13992 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:00:53.348423 13992 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790395095474643-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790395095474643-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34813
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790395095474643-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:00:53.348729 13992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:00:53.348968 13992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:00:53.353704 13997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:00:53.353894 13998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:00:53.353776 14000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:00:53.353922 13992 server_base.cc:1034] running on GCE node
I20260926 04:00:53.354184 13992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:00:53.354490 13992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:00:53.355654 13992 hybrid_clock.cc:648] HybridClock initialized: now 1790395253355644 us; error 42 us; skew 500 ppm
Sep 26 04:00:53 dist-test-slave-nhpw krb5kdc[13979](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395253, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:00:53.358208 13992 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 04:00:53.358712 13992 webserver.cc:460] Webserver started at http://127.0.98.62:38543/ using document root <none> and password file <none>
I20260926 04:00:53.358942 13992 fs_manager.cc:359] Metadata directory not provided
I20260926 04:00:53.358987 13992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:00:53.359129 13992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:00:53.360208 13992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790395095474643-392-0/minicluster-data/master-0/data/instance:
uuid: "76d64e6f7c00409ba5b7d32afdaa5ec7"
format_stamp: "Formatted at 2026-09-26 04:00:53 on dist-test-slave-nhpw"
server_key: "7cd0fb127ed53a3f52b23a8356ed41d2"
server_key_iv: "3a446ad72b2fca6cb11e5fa68ae01187"
server_key_version: "encryptionkey@0"
I20260926 04:00:53.360582 13992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790395095474643-392-0/minicluster-data/master-0/wal/instance:
uuid: "76d64e6f7c00409ba5b7d32afdaa5ec7"
format_stamp: "Formatted at 2026-09-26 04:00:53 on dist-test-slave-nhpw"
server_key: "7cd0fb127ed53a3f52b23a8356ed41d2"
server_key_iv: "3a446ad72b2fca6cb11e5fa68ae01187"
server_key_version: "encryptionkey@0"
I20260926 04:00:53.361909 13992 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:00:53.362692 14007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:00:53.362895 13992 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:00:53.362993 13992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790395095474643-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790395095474643-392-0/minicluster-data/master-0/wal
uuid: "76d64e6f7c00409ba5b7d32afdaa5ec7"
format_stamp: "Formatted at 2026-09-26 04:00:53 on dist-test-slave-nhpw"
server_key: "7cd0fb127ed53a3f52b23a8356ed41d2"
server_key_iv: "3a446ad72b2fca6cb11e5fa68ae01187"
server_key_version: "encryptionkey@0"
I20260926 04:00:53.363080 13992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790395095474643-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790395095474643-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790395095474643-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:00:53.377651 13992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:00:53.379982 13992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:00:53.380151 13992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:00:53.384099 13992 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34813
I20260926 04:00:53.384179 14059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34813 every 8 connection(s)
I20260926 04:00:53.384585 13992 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
I20260926 04:00:53.385494 14060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:00:53.387799 14060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76d64e6f7c00409ba5b7d32afdaa5ec7: Bootstrap starting.
I20260926 04:00:53.388389 14060 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76d64e6f7c00409ba5b7d32afdaa5ec7: Neither blocks nor log segments found. Creating new log.
I20260926 04:00:53.388746 14060 log.cc:824] T 00000000000000000000000000000000 P 76d64e6f7c00409ba5b7d32afdaa5ec7: Log is configured to *not* fsync() on all Append() calls
I20260926 04:00:53.388978 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 13992
I20260926 04:00:53.389078 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790395095474643-392-0/minicluster-data/master-0/wal/instance
I20260926 04:00:53.389264 392 external_mini_cluster.cc:1428] Setting key 56fad13854ff1015789810a97cc76bf8
I20260926 04:00:53.389524 14060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76d64e6f7c00409ba5b7d32afdaa5ec7: No bootstrap required, opened a new log
I20260926 04:00:53.391649 14060 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76d64e6f7c00409ba5b7d32afdaa5ec7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76d64e6f7c00409ba5b7d32afdaa5ec7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34813 } }
I20260926 04:00:53.391825 14060 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76d64e6f7c00409ba5b7d32afdaa5ec7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:00:53.391858 14060 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76d64e6f7c00409ba5b7d32afdaa5ec7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76d64e6f7c00409ba5b7d32afdaa5ec7, State: Initialized, Role: FOLLOWER
I20260926 04:00:53.392012 14060 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76d64e6f7c00409ba5b7d32afdaa5ec7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76d64e6f7c00409ba5b7d32afdaa5ec7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34813 } }
I20260926 04:00:53.392099 14060 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76d64e6f7c00409ba5b7d32afdaa5ec7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:00:53.392140 14060 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76d64e6f7c00409ba5b7d32afdaa5ec7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:00:53.392185 14060 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76d64e6f7c00409ba5b7d32afdaa5ec7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:00:53.392891 14060 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76d64e6f7c00409ba5b7d32afdaa5ec7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76d64e6f7c00409ba5b7d32afdaa5ec7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34813 } }
I20260926 04:00:53.392995 14060 leader_election.cc:302] T 00000000000000000000000000000000 P 76d64e6f7c00409ba5b7d32afdaa5ec7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76d64e6f7c00409ba5b7d32afdaa5ec7; no voters:
I20260926 04:00:53.393149 14060 leader_election.cc:288] T 00000000000000000000000000000000 P 76d64e6f7c00409ba5b7d32afdaa5ec7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 04:00:53.393217 14065 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76d64e6f7c00409ba5b7d32afdaa5ec7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:00:53.393391 14060 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76d64e6f7c00409ba5b7d32afdaa5ec7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:00:53.393397 14065 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76d64e6f7c00409ba5b7d32afdaa5ec7 [term 1 LEADER]: Becoming Leader. State: Replica: 76d64e6f7c00409ba5b7d32afdaa5ec7, State: Running, Role: LEADER
I20260926 04:00:53.394295 14065 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76d64e6f7c00409ba5b7d32afdaa5ec7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76d64e6f7c00409ba5b7d32afdaa5ec7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34813 } }
I20260926 04:00:53.394905 14066 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76d64e6f7c00409ba5b7d32afdaa5ec7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76d64e6f7c00409ba5b7d32afdaa5ec7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76d64e6f7c00409ba5b7d32afdaa5ec7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34813 } } }
I20260926 04:00:53.395642 14066 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76d64e6f7c00409ba5b7d32afdaa5ec7 [sys.catalog]: This master's current role is: LEADER
I20260926 04:00:53.395041 14067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76d64e6f7c00409ba5b7d32afdaa5ec7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76d64e6f7c00409ba5b7d32afdaa5ec7. Latest consensus state: current_term: 1 leader_uuid: "76d64e6f7c00409ba5b7d32afdaa5ec7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76d64e6f7c00409ba5b7d32afdaa5ec7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34813 } } }
I20260926 04:00:53.396744 14067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76d64e6f7c00409ba5b7d32afdaa5ec7 [sys.catalog]: This master's current role is: LEADER
W20260926 04:00:53.395251 14079 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 76d64e6f7c00409ba5b7d32afdaa5ec7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 04:00:53.396824 14079 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 04:00:53.397079 14083 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:00:53.397296 14083 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 04:00:53 dist-test-slave-nhpw krb5kdc[13979](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395253, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 04:00:53.398988 14083 catalog_manager.cc:1347] Generated new cluster ID: 37a2d126212d4f99a9f3ffdf07ef5134
I20260926 04:00:53.399037 14083 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 04:00:53.401718 14064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:53.389760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55482 (local address 127.0.98.62:34813)
0926 04:00:53.389977 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 04:00:53.389986 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 04:00:53.390021 (+ 35us) server_negotiation.cc:407] Connection header received
0926 04:00:53.390848 (+ 827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:00:53.390858 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:00:53.390920 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:00:53.391073 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:00:53.391659 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:53.393362 (+ 1703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:53.394158 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:53.394873 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:53.394977 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:53.397984 (+ 3007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:00:53.397989 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:00:53.397989 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:00:53.398001 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:00:53.400056 (+ 2055us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:00:53.400483 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:00:53.400485 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:00:53.400485 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:00:53.400583 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:00:53.400824 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:00:53.400824 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:00:53.400825 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:00:53.401105 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:00:53.401242 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 04:00:53.401394 (+ 152us) server_negotiation.cc:299] Negotiation successful
0926 04:00:53.401521 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":103,"threads_started":1}
I20260926 04:00:53.402907 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 04:00:53.416075 14083 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:00:53.416891 14083 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:00:53.425781 14083 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76d64e6f7c00409ba5b7d32afdaa5ec7: Generated new TSK 0
I20260926 04:00:53.426002 14083 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 04:00:53.456945 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 13992
2026-09-26T04:00:53Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1399 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 04:00:53 dist-test-slave-nhpw krb5kdc[14088](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 04:00:53 dist-test-slave-nhpw krb5kdc[14088](info): set up 2 sockets
Sep 26 04:00:53 dist-test-slave-nhpw krb5kdc[14088](info): commencing operation
krb5kdc: starting...
W20260926 04:00:54.488828 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/krb5kdc/test-user.keytab.
Sep 26 04:00:54 dist-test-slave-nhpw krb5kdc[14088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T04:00:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:00:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 04:00:54.605854 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46215
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46215
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:00:54.687688 14101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:00:54.687947 14101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:00:54.687985 14101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:00:54.689173 14101 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:00:54.689244 14101 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:00:54.689270 14101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:00:54.689291 14101 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:00:54.689318 14101 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:00:54.690800 14101 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46215
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14101
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:00:54.691084 14101 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:00:54.691309 14101 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:00:54.696233 14106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:00:54.696316 14101 server_base.cc:1034] running on GCE node
W20260926 04:00:54.696403 14109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:00:54.696240 14107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:00:54.696774 14101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:00:54.697026 14101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:00:54.698203 14101 hybrid_clock.cc:648] HybridClock initialized: now 1790395254698184 us; error 37 us; skew 500 ppm
Sep 26 04:00:54 dist-test-slave-nhpw krb5kdc[14088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395254, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:00:54.700838 14101 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 04:00:54.701380 14101 webserver.cc:460] Webserver started at http://127.0.98.62:34903/ using document root <none> and password file <none>
I20260926 04:00:54.701570 14101 fs_manager.cc:359] Metadata directory not provided
I20260926 04:00:54.701611 14101 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:00:54.701700 14101 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:00:54.702703 14101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/master-0/data/instance:
uuid: "a2e82428eba746a8b01d279a4f8b659f"
format_stamp: "Formatted at 2026-09-26 04:00:54 on dist-test-slave-nhpw"
server_key: "88b3ac9de17d77ea99a6433a2d34f8e7"
server_key_iv: "b1352df44537b6a5c96335f7bf538c1c"
server_key_version: "encryptionkey@0"
I20260926 04:00:54.702998 14101 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/master-0/wal/instance:
uuid: "a2e82428eba746a8b01d279a4f8b659f"
format_stamp: "Formatted at 2026-09-26 04:00:54 on dist-test-slave-nhpw"
server_key: "88b3ac9de17d77ea99a6433a2d34f8e7"
server_key_iv: "b1352df44537b6a5c96335f7bf538c1c"
server_key_version: "encryptionkey@0"
I20260926 04:00:54.704222 14101 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 04:00:54.705008 14116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:00:54.705267 14101 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 04:00:54.705377 14101 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/master-0/wal
uuid: "a2e82428eba746a8b01d279a4f8b659f"
format_stamp: "Formatted at 2026-09-26 04:00:54 on dist-test-slave-nhpw"
server_key: "88b3ac9de17d77ea99a6433a2d34f8e7"
server_key_iv: "b1352df44537b6a5c96335f7bf538c1c"
server_key_version: "encryptionkey@0"
I20260926 04:00:54.705469 14101 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:00:54.718034 14101 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:00:54.720404 14101 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:00:54.720573 14101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:00:54.724356 14101 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46215
I20260926 04:00:54.724416 14168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46215 every 8 connection(s)
I20260926 04:00:54.724793 14101 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
I20260926 04:00:54.725797 14169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:00:54.727985 14169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2e82428eba746a8b01d279a4f8b659f: Bootstrap starting.
I20260926 04:00:54.728510 14169 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2e82428eba746a8b01d279a4f8b659f: Neither blocks nor log segments found. Creating new log.
I20260926 04:00:54.728845 14169 log.cc:824] T 00000000000000000000000000000000 P a2e82428eba746a8b01d279a4f8b659f: Log is configured to *not* fsync() on all Append() calls
I20260926 04:00:54.729540 14169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2e82428eba746a8b01d279a4f8b659f: No bootstrap required, opened a new log
I20260926 04:00:54.730450 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 14101
I20260926 04:00:54.730553 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/master-0/wal/instance
I20260926 04:00:54.730742 392 external_mini_cluster.cc:1428] Setting key a29986b7cb575dc0b38c6910071ed2cd
I20260926 04:00:54.731688 14169 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2e82428eba746a8b01d279a4f8b659f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2e82428eba746a8b01d279a4f8b659f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46215 } }
I20260926 04:00:54.731861 14169 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2e82428eba746a8b01d279a4f8b659f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:00:54.731902 14169 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2e82428eba746a8b01d279a4f8b659f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e82428eba746a8b01d279a4f8b659f, State: Initialized, Role: FOLLOWER
I20260926 04:00:54.732043 14169 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2e82428eba746a8b01d279a4f8b659f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2e82428eba746a8b01d279a4f8b659f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46215 } }
I20260926 04:00:54.732129 14169 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2e82428eba746a8b01d279a4f8b659f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:00:54.732172 14169 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2e82428eba746a8b01d279a4f8b659f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:00:54.732225 14169 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2e82428eba746a8b01d279a4f8b659f [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:00:54.733219 14169 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2e82428eba746a8b01d279a4f8b659f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2e82428eba746a8b01d279a4f8b659f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46215 } }
I20260926 04:00:54.733309 14169 leader_election.cc:302] T 00000000000000000000000000000000 P a2e82428eba746a8b01d279a4f8b659f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2e82428eba746a8b01d279a4f8b659f; no voters:
I20260926 04:00:54.733523 14169 leader_election.cc:288] T 00000000000000000000000000000000 P a2e82428eba746a8b01d279a4f8b659f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 04:00:54.733543 14174 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2e82428eba746a8b01d279a4f8b659f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:00:54.733654 14174 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2e82428eba746a8b01d279a4f8b659f [term 1 LEADER]: Becoming Leader. State: Replica: a2e82428eba746a8b01d279a4f8b659f, State: Running, Role: LEADER
I20260926 04:00:54.733789 14169 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2e82428eba746a8b01d279a4f8b659f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:00:54.733793 14174 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2e82428eba746a8b01d279a4f8b659f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2e82428eba746a8b01d279a4f8b659f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46215 } }
I20260926 04:00:54.734310 14176 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2e82428eba746a8b01d279a4f8b659f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2e82428eba746a8b01d279a4f8b659f. Latest consensus state: current_term: 1 leader_uuid: "a2e82428eba746a8b01d279a4f8b659f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2e82428eba746a8b01d279a4f8b659f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46215 } } }
I20260926 04:00:54.734403 14176 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2e82428eba746a8b01d279a4f8b659f [sys.catalog]: This master's current role is: LEADER
I20260926 04:00:54.734607 14175 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2e82428eba746a8b01d279a4f8b659f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2e82428eba746a8b01d279a4f8b659f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2e82428eba746a8b01d279a4f8b659f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46215 } } }
I20260926 04:00:54.734865 14175 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2e82428eba746a8b01d279a4f8b659f [sys.catalog]: This master's current role is: LEADER
I20260926 04:00:54.735037 14183 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:00:54.735584 14183 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:00:54.737123 14183 catalog_manager.cc:1347] Generated new cluster ID: 8d7e2452f04944cb98926530d868de23
I20260926 04:00:54.737171 14183 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 04:00:54 dist-test-slave-nhpw krb5kdc[14088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 04:00:54.741866 14183 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:00:54.742383 14183 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:00:54.742518 14173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:54.731259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45904 (local address 127.0.98.62:46215)
0926 04:00:54.731442 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 04:00:54.731457 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 04:00:54.731484 (+ 27us) server_negotiation.cc:407] Connection header received
0926 04:00:54.731649 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:00:54.731654 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:00:54.731694 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:00:54.731811 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:00:54.732673 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:54.733705 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:54.734716 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:54.735334 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:54.735942 (+ 608us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:54.738281 (+ 2339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:00:54.738285 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:00:54.738285 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:00:54.738297 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:00:54.740245 (+ 1948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:00:54.740681 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:00:54.740684 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:00:54.740684 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:00:54.740825 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:00:54.741052 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:00:54.741052 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:00:54.741053 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:00:54.741341 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:00:54.741949 (+ 608us) server_negotiation.cc:1035] Waiting for connection context
0926 04:00:54.742020 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 04:00:54.742157 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":97,"threads_started":1}
I20260926 04:00:54.746369 14183 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2e82428eba746a8b01d279a4f8b659f: Generated new TSK 0
I20260926 04:00:54.746541 14183 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 04:00:54.773414 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46215
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:00:54.852775 14195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:00:54.853003 14195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:00:54.853039 14195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:00:54.854194 14195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:00:54.854274 14195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:00:54.854321 14195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 04:00:54.855748 14195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:00:54.856007 14195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:00:54.856235 14195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:00:54.861068 14201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:00:54.861148 14203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:00:54.861064 14200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:00:54.861289 14195 server_base.cc:1034] running on GCE node
I20260926 04:00:54.861475 14195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:00:54.861673 14195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:00:54.862809 14195 hybrid_clock.cc:648] HybridClock initialized: now 1790395254862803 us; error 31 us; skew 500 ppm
Sep 26 04:00:54 dist-test-slave-nhpw krb5kdc[14088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395254, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:00:54.865273 14195 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 04:00:54.865674 14195 webserver.cc:460] Webserver started at http://127.0.98.1:34091/ using document root <none> and password file <none>
I20260926 04:00:54.865903 14195 fs_manager.cc:359] Metadata directory not provided
I20260926 04:00:54.865947 14195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:00:54.866106 14195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:00:54.867102 14195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-0/data/instance:
uuid: "c199a3d5f00d4157bb393acf090b3fda"
format_stamp: "Formatted at 2026-09-26 04:00:54 on dist-test-slave-nhpw"
server_key: "13f7f460d5365dcbd0448ac676ca5d87"
server_key_iv: "75b9a57771d10f436ee080c912d88320"
server_key_version: "encryptionkey@0"
I20260926 04:00:54.867445 14195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c199a3d5f00d4157bb393acf090b3fda"
format_stamp: "Formatted at 2026-09-26 04:00:54 on dist-test-slave-nhpw"
server_key: "13f7f460d5365dcbd0448ac676ca5d87"
server_key_iv: "75b9a57771d10f436ee080c912d88320"
server_key_version: "encryptionkey@0"
I20260926 04:00:54.868748 14195 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 04:00:54.869462 14210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:00:54.869642 14195 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 04:00:54.869729 14195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-0/wal
uuid: "c199a3d5f00d4157bb393acf090b3fda"
format_stamp: "Formatted at 2026-09-26 04:00:54 on dist-test-slave-nhpw"
server_key: "13f7f460d5365dcbd0448ac676ca5d87"
server_key_iv: "75b9a57771d10f436ee080c912d88320"
server_key_version: "encryptionkey@0"
I20260926 04:00:54.869810 14195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:00:54.878710 14195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:00:54.880972 14195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:00:54.881126 14195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:00:54.881350 14195 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:00:54.881634 14195 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:00:54.881687 14195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:00:54.881727 14195 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:00:54.881755 14195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:00:54.887297 14195 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37653
I20260926 04:00:54.887396 14323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37653 every 8 connection(s)
I20260926 04:00:54.887663 14195 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
I20260926 04:00:54.888017 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 14195
I20260926 04:00:54.888120 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-0/wal/instance
I20260926 04:00:54.888322 392 external_mini_cluster.cc:1428] Setting key 39ddde4aff1c77e1fa6ea0ec5ce077ad
Sep 26 04:00:54 dist-test-slave-nhpw krb5kdc[14088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395254, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 04:00:54.897156 14173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:54.888382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58939 (local address 127.0.98.62:46215)
0926 04:00:54.888435 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 04:00:54.888437 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:00:54.888622 (+ 185us) server_negotiation.cc:407] Connection header received
0926 04:00:54.889261 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:00:54.889263 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:00:54.889279 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:00:54.889346 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:00:54.890266 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:54.890774 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:54.891779 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:54.892001 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:54.892087 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:54.894335 (+ 2248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:00:54.894337 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:00:54.894337 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:00:54.894346 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:00:54.895693 (+ 1347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:00:54.896183 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:00:54.896184 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:00:54.896184 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:00:54.896248 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:00:54.896542 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:00:54.896542 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:00:54.896542 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:00:54.896705 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:00:54.896757 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 04:00:54.897009 (+ 252us) server_negotiation.cc:299] Negotiation successful
0926 04:00:54.897079 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 04:00:54.897323 14326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:54.888371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.98.1:58939)
0926 04:00:54.888517 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 04:00:54.888556 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 04:00:54.889163 (+ 607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:00:54.889399 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:00:54.889403 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:00:54.889938 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:00:54.890170 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:00:54.890175 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:00:54.890877 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:00:54.890879 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:00:54.891665 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:00:54.891667 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:00:54.892098 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:00:54.892099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:00:54.892226 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:54.892764 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:00:54.892773 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:00:54.894238 (+ 1465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:00:54.895776 (+ 1538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:00:54.895784 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:00:54.895786 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:00:54.896082 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:00:54.896319 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:00:54.896320 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:00:54.896320 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:00:54.896473 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:00:54.896767 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:00:54.896773 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:00:54.896895 (+ 122us) client_negotiation.cc:769] Sending connection context
0926 04:00:54.896982 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 04:00:54.897104 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":80,"threads_started":1}
I20260926 04:00:54.897926 14324 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46215
I20260926 04:00:54.898006 14324 heartbeater.cc:464] Registering TS with master...
I20260926 04:00:54.898145 14324 heartbeater.cc:511] Master 127.0.98.62:46215 requested a full tablet report, sending...
I20260926 04:00:54.898691 14133 ts_manager.cc:194] Registered new tserver with Master: c199a3d5f00d4157bb393acf090b3fda (127.0.98.1:37653)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260926 04:00:54.899510 14133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58939
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 04:00:54.913580 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46215
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:00:54.993595 14329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:00:54.993808 14329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:00:54.993832 14329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:00:54.994967 14329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:00:54.995003 14329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:00:54.995029 14329 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 04:00:54.996402 14329 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14329
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:00:54.996587 14329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:00:54.996891 14329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:00:55.002182 14334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:00:55.002182 14335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:00:55.002272 14337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:00:55.002936 14329 server_base.cc:1034] running on GCE node
I20260926 04:00:55.003142 14329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:00:55.003387 14329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:00:55.004541 14329 hybrid_clock.cc:648] HybridClock initialized: now 1790395255004525 us; error 30 us; skew 500 ppm
Sep 26 04:00:55 dist-test-slave-nhpw krb5kdc[14088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395255, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:00:55.007517 14329 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 04:00:55.008011 14329 webserver.cc:460] Webserver started at http://127.0.98.2:40201/ using document root <none> and password file <none>
I20260926 04:00:55.008286 14329 fs_manager.cc:359] Metadata directory not provided
I20260926 04:00:55.008363 14329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:00:55.008502 14329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:00:55.009676 14329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-1/data/instance:
uuid: "a9e9f973c5bf41f0bc6967ce64cc9706"
format_stamp: "Formatted at 2026-09-26 04:00:55 on dist-test-slave-nhpw"
server_key: "0bdff880fe199481e5f395441305c8ea"
server_key_iv: "5ec0f94efb527d4d81c2fc88a306ae2c"
server_key_version: "encryptionkey@0"
I20260926 04:00:55.010058 14329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a9e9f973c5bf41f0bc6967ce64cc9706"
format_stamp: "Formatted at 2026-09-26 04:00:55 on dist-test-slave-nhpw"
server_key: "0bdff880fe199481e5f395441305c8ea"
server_key_iv: "5ec0f94efb527d4d81c2fc88a306ae2c"
server_key_version: "encryptionkey@0"
I20260926 04:00:55.011467 14329 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 04:00:55.012334 14344 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:00:55.012686 14329 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:00:55.012799 14329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-1/wal
uuid: "a9e9f973c5bf41f0bc6967ce64cc9706"
format_stamp: "Formatted at 2026-09-26 04:00:55 on dist-test-slave-nhpw"
server_key: "0bdff880fe199481e5f395441305c8ea"
server_key_iv: "5ec0f94efb527d4d81c2fc88a306ae2c"
server_key_version: "encryptionkey@0"
I20260926 04:00:55.012887 14329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:00:55.025830 14329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:00:55.027963 14329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:00:55.028131 14329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:00:55.028368 14329 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:00:55.028751 14329 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:00:55.028803 14329 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:00:55.028849 14329 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:00:55.028877 14329 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:00:55.034842 14329 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35223
I20260926 04:00:55.034896 14457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35223 every 8 connection(s)
I20260926 04:00:55.035251 14329 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
I20260926 04:00:55.037968 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 14329
I20260926 04:00:55.038082 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-1/wal/instance
I20260926 04:00:55.038285 392 external_mini_cluster.cc:1428] Setting key 21f5d2aad433beabcfd9bf6e392fe2c0
Sep 26 04:00:55 dist-test-slave-nhpw krb5kdc[14088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395255, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260926 04:00:55.046495 14173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:55.036024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37501 (local address 127.0.98.62:46215)
0926 04:00:55.036146 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 04:00:55.036148 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:00:55.036445 (+ 297us) server_negotiation.cc:407] Connection header received
0926 04:00:55.037038 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:00:55.037039 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:00:55.037050 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:00:55.037107 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:00:55.038264 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:55.038940 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:55.040512 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:55.040809 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:55.040845 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:55.043477 (+ 2632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:00:55.043478 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:00:55.043479 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:00:55.043486 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:00:55.045020 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:00:55.045461 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:00:55.045462 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:00:55.045463 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:00:55.045528 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:00:55.045796 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:00:55.045797 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:00:55.045797 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:00:55.045916 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:00:55.045987 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 04:00:55.046300 (+ 313us) server_negotiation.cc:299] Negotiation successful
0926 04:00:55.046404 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260926 04:00:55.046660 14460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:55.035993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.98.2:37501)
0926 04:00:55.036312 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0926 04:00:55.036350 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 04:00:55.036940 (+ 590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:00:55.037205 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:00:55.037212 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:00:55.037817 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:00:55.038118 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:00:55.038125 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:00:55.039042 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:00:55.039043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:00:55.040394 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:00:55.040398 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:00:55.040921 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:00:55.040921 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:00:55.041019 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:55.041655 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:00:55.041673 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:00:55.043358 (+ 1685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:00:55.045120 (+ 1762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:00:55.045127 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:00:55.045130 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:00:55.045366 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:00:55.045601 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:00:55.045602 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:00:55.045603 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:00:55.045717 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:00:55.045990 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:00:55.045997 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:00:55.046161 (+ 164us) client_negotiation.cc:769] Sending connection context
0926 04:00:55.046267 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 04:00:55.046426 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":182,"threads_started":1}
I20260926 04:00:55.047223 14458 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46215
I20260926 04:00:55.047335 14458 heartbeater.cc:464] Registering TS with master...
I20260926 04:00:55.047529 14458 heartbeater.cc:511] Master 127.0.98.62:46215 requested a full tablet report, sending...
I20260926 04:00:55.047931 14133 ts_manager.cc:194] Registered new tserver with Master: a9e9f973c5bf41f0bc6967ce64cc9706 (127.0.98.2:35223)
I20260926 04:00:55.048424 14133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37501
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 04:00:55.064092 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46215
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:00:55.145038 14463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:00:55.145241 14463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:00:55.145262 14463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:00:55.146344 14463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:00:55.146376 14463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:00:55.146401 14463 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 04:00:55.147742 14463 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:00:55.147986 14463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:00:55.148229 14463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:00:55.153409 14471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:00:55.153415 14469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:00:55.153419 14468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:00:55.153589 14463 server_base.cc:1034] running on GCE node
I20260926 04:00:55.153913 14463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:00:55.154181 14463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:00:55.155344 14463 hybrid_clock.cc:648] HybridClock initialized: now 1790395255155335 us; error 41 us; skew 500 ppm
Sep 26 04:00:55 dist-test-slave-nhpw krb5kdc[14088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395255, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:00:55.157859 14463 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 04:00:55.158334 14463 webserver.cc:460] Webserver started at http://127.0.98.3:42519/ using document root <none> and password file <none>
I20260926 04:00:55.158571 14463 fs_manager.cc:359] Metadata directory not provided
I20260926 04:00:55.158638 14463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:00:55.158784 14463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:00:55.159789 14463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-2/data/instance:
uuid: "febbc732bc55422e9ee6bdf4c48eec3a"
format_stamp: "Formatted at 2026-09-26 04:00:55 on dist-test-slave-nhpw"
server_key: "1c1cff0e74cb94c8d1182ad91ac5801f"
server_key_iv: "35d44f9bc936126682151ef75e1c830d"
server_key_version: "encryptionkey@0"
I20260926 04:00:55.160141 14463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-2/wal/instance:
uuid: "febbc732bc55422e9ee6bdf4c48eec3a"
format_stamp: "Formatted at 2026-09-26 04:00:55 on dist-test-slave-nhpw"
server_key: "1c1cff0e74cb94c8d1182ad91ac5801f"
server_key_iv: "35d44f9bc936126682151ef75e1c830d"
server_key_version: "encryptionkey@0"
I20260926 04:00:55.161537 14463 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 04:00:55.162258 14478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:00:55.162428 14463 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 04:00:55.162524 14463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-2/wal
uuid: "febbc732bc55422e9ee6bdf4c48eec3a"
format_stamp: "Formatted at 2026-09-26 04:00:55 on dist-test-slave-nhpw"
server_key: "1c1cff0e74cb94c8d1182ad91ac5801f"
server_key_iv: "35d44f9bc936126682151ef75e1c830d"
server_key_version: "encryptionkey@0"
I20260926 04:00:55.162604 14463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:00:55.186743 14463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:00:55.188818 14463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:00:55.189003 14463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:00:55.189246 14463 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:00:55.189560 14463 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:00:55.189617 14463 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:00:55.189657 14463 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:00:55.189699 14463 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:00:55.195441 14463 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37355
I20260926 04:00:55.195488 14591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37355 every 8 connection(s)
I20260926 04:00:55.195809 14463 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
I20260926 04:00:55.198694 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 14463
I20260926 04:00:55.198793 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-2/wal/instance
I20260926 04:00:55.198961 392 external_mini_cluster.cc:1428] Setting key 3636d5245ee1bee2fb3200f330efaa35
Sep 26 04:00:55 dist-test-slave-nhpw krb5kdc[14088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395255, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 04:00:55.205668 14173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:55.196540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55971 (local address 127.0.98.62:46215)
0926 04:00:55.196606 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 04:00:55.196608 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:00:55.196816 (+ 208us) server_negotiation.cc:407] Connection header received
0926 04:00:55.197297 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:00:55.197298 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:00:55.197307 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:00:55.197381 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:00:55.198275 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:55.199078 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:55.200348 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:55.200537 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:55.200599 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:55.202848 (+ 2249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:00:55.202849 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:00:55.202850 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:00:55.202858 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:00:55.204044 (+ 1186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:00:55.204516 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:00:55.204517 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:00:55.204517 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:00:55.204587 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:00:55.204991 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:00:55.204992 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:00:55.204993 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:00:55.205119 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:00:55.205200 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 04:00:55.205480 (+ 280us) server_negotiation.cc:299] Negotiation successful
0926 04:00:55.205563 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 04:00:55.205860 14594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:55.196499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.98.3:55971)
0926 04:00:55.196687 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 04:00:55.196729 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 04:00:55.197212 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:00:55.197443 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:00:55.197446 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:00:55.198003 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:00:55.198198 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:00:55.198202 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:00:55.199180 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:00:55.199181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:00:55.200246 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:00:55.200254 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:00:55.200614 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:00:55.200616 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:00:55.200757 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:55.201307 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:00:55.201316 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:00:55.202744 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:00:55.204173 (+ 1429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:00:55.204180 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:00:55.204182 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:00:55.204434 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:00:55.204751 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:00:55.204752 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:00:55.204752 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:00:55.204918 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:00:55.205191 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:00:55.205197 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:00:55.205344 (+ 147us) client_negotiation.cc:769] Sending connection context
0926 04:00:55.205457 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 04:00:55.205621 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":92,"threads_started":1}
I20260926 04:00:55.206333 14592 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46215
I20260926 04:00:55.206439 14592 heartbeater.cc:464] Registering TS with master...
I20260926 04:00:55.206636 14592 heartbeater.cc:511] Master 127.0.98.62:46215 requested a full tablet report, sending...
I20260926 04:00:55.207042 14133 ts_manager.cc:194] Registered new tserver with Master: febbc732bc55422e9ee6bdf4c48eec3a (127.0.98.3:37355)
I20260926 04:00:55.207419 14133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55971
I20260926 04:00:55.211024 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 04:00:55.218571 14173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:55.212416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45912 (local address 127.0.98.62:46215)
0926 04:00:55.212478 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 04:00:55.212480 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:00:55.212587 (+ 107us) server_negotiation.cc:407] Connection header received
0926 04:00:55.212717 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:00:55.212718 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:00:55.212728 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:00:55.212785 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:00:55.213369 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:55.213849 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:55.214586 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:55.214782 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:55.214856 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:55.216073 (+ 1217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:00:55.216075 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:00:55.216075 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:00:55.216082 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:00:55.217368 (+ 1286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:00:55.217735 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:00:55.217736 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:00:55.217737 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:00:55.217792 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:00:55.217986 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:00:55.217987 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:00:55.217987 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:00:55.218146 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:00:55.218251 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 04:00:55.218394 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 04:00:55.218469 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 04:00:55.219997 14133 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45912:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 04:00:55.221668 14133 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 04:00:55.227687 14602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:55.223729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37355 (local address 127.0.0.1:45832)
0926 04:00:55.224158 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0926 04:00:55.224168 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 04:00:55.224234 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:00:55.224440 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:00:55.224443 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:00:55.224445 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:00:55.224507 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:00:55.224509 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:00:55.225487 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:00:55.225488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:00:55.226551 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:00:55.226554 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:00:55.227382 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:00:55.227383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:00:55.227461 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:55.227474 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 04:00:55.227550 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 04:00:55.227596 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"mutex_wait_us":4,"spinlock_wait_cycles":768,"thread_start_us":55,"threads_started":1}
I20260926 04:00:55.227700 14603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:55.223888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45832 (local address 127.0.98.3:37355)
0926 04:00:55.224056 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 04:00:55.224061 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 04:00:55.224227 (+ 166us) server_negotiation.cc:407] Connection header received
0926 04:00:55.224300 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:00:55.224303 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:00:55.224319 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:00:55.224425 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:00:55.224589 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:55.225365 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:55.226624 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:55.227291 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:55.227366 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:55.227420 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 04:00:55.227576 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 04:00:55.227617 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":93,"threads_started":1}
I20260926 04:00:55.228466 14601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:55.223729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35223 (local address 127.0.0.1:54862)
0926 04:00:55.224080 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0926 04:00:55.224117 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 04:00:55.224217 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:00:55.224615 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:00:55.224615 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:00:55.224617 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:00:55.224688 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:00:55.224689 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:00:55.226035 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:00:55.226036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:00:55.226990 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:00:55.226992 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:00:55.228233 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:00:55.228234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:00:55.228299 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:55.228303 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 04:00:55.228358 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 04:00:55.228401 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":62,"threads_started":1}
I20260926 04:00:55.228439 14526 tablet_service.cc:1467] Processing CreateTablet for tablet 7aba625221014e3f8069dddce55382d1 (DEFAULT_TABLE table=test-table [id=cf25aa26c03f48a19dbc9cb2bd83c91d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:00:55.228729 14604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:55.223909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54862 (local address 127.0.98.2:35223)
0926 04:00:55.224095 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 04:00:55.224100 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 04:00:55.224204 (+ 104us) server_negotiation.cc:407] Connection header received
0926 04:00:55.224300 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:00:55.224303 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:00:55.224321 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:00:55.224413 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:00:55.224772 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:55.225924 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:55.227074 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:55.228130 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:55.228229 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:55.228283 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 04:00:55.228399 (+ 116us) server_negotiation.cc:299] Negotiation successful
0926 04:00:55.228439 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":78,"threads_started":1}
I20260926 04:00:55.228853 14526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7aba625221014e3f8069dddce55382d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:00:55.229251 14600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:55.223747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37653 (local address 127.0.0.1:60648)
0926 04:00:55.224321 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0926 04:00:55.224330 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 04:00:55.224394 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:00:55.224682 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:00:55.224682 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:00:55.224684 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:00:55.224721 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:00:55.224722 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:00:55.227062 (+ 2340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:00:55.227063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:00:55.227970 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:00:55.227972 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:00:55.228995 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:00:55.228996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:00:55.229054 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:55.229057 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 04:00:55.229106 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 04:00:55.229155 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":92,"threads_started":1}
I20260926 04:00:55.229535 14605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:55.223984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60648 (local address 127.0.98.1:37653)
0926 04:00:55.224185 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 04:00:55.224189 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 04:00:55.224457 (+ 268us) server_negotiation.cc:407] Connection header received
0926 04:00:55.224503 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:00:55.224505 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:00:55.224521 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:00:55.224589 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:00:55.224798 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:55.225933 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:55.228083 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:55.228832 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:55.228904 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:55.228958 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 04:00:55.229412 (+ 454us) server_negotiation.cc:299] Negotiation successful
0926 04:00:55.229458 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":75,"threads_started":1}
I20260926 04:00:55.229641 14392 tablet_service.cc:1467] Processing CreateTablet for tablet 7aba625221014e3f8069dddce55382d1 (DEFAULT_TABLE table=test-table [id=cf25aa26c03f48a19dbc9cb2bd83c91d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:00:55.229882 14392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7aba625221014e3f8069dddce55382d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:00:55.230116 14258 tablet_service.cc:1467] Processing CreateTablet for tablet 7aba625221014e3f8069dddce55382d1 (DEFAULT_TABLE table=test-table [id=cf25aa26c03f48a19dbc9cb2bd83c91d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:00:55.230443 14258 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7aba625221014e3f8069dddce55382d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:00:55.232333 14606 tablet_bootstrap.cc:492] T 7aba625221014e3f8069dddce55382d1 P febbc732bc55422e9ee6bdf4c48eec3a: Bootstrap starting.
I20260926 04:00:55.232668 14607 tablet_bootstrap.cc:492] T 7aba625221014e3f8069dddce55382d1 P a9e9f973c5bf41f0bc6967ce64cc9706: Bootstrap starting.
I20260926 04:00:55.233075 14606 tablet_bootstrap.cc:654] T 7aba625221014e3f8069dddce55382d1 P febbc732bc55422e9ee6bdf4c48eec3a: Neither blocks nor log segments found. Creating new log.
I20260926 04:00:55.233281 14607 tablet_bootstrap.cc:654] T 7aba625221014e3f8069dddce55382d1 P a9e9f973c5bf41f0bc6967ce64cc9706: Neither blocks nor log segments found. Creating new log.
I20260926 04:00:55.233706 14608 tablet_bootstrap.cc:492] T 7aba625221014e3f8069dddce55382d1 P c199a3d5f00d4157bb393acf090b3fda: Bootstrap starting.
I20260926 04:00:55.233767 14606 log.cc:824] T 7aba625221014e3f8069dddce55382d1 P febbc732bc55422e9ee6bdf4c48eec3a: Log is configured to *not* fsync() on all Append() calls
I20260926 04:00:55.233855 14607 log.cc:824] T 7aba625221014e3f8069dddce55382d1 P a9e9f973c5bf41f0bc6967ce64cc9706: Log is configured to *not* fsync() on all Append() calls
I20260926 04:00:55.234412 14608 tablet_bootstrap.cc:654] T 7aba625221014e3f8069dddce55382d1 P c199a3d5f00d4157bb393acf090b3fda: Neither blocks nor log segments found. Creating new log.
I20260926 04:00:55.234607 14607 tablet_bootstrap.cc:492] T 7aba625221014e3f8069dddce55382d1 P a9e9f973c5bf41f0bc6967ce64cc9706: No bootstrap required, opened a new log
I20260926 04:00:55.234683 14607 ts_tablet_manager.cc:1397] T 7aba625221014e3f8069dddce55382d1 P a9e9f973c5bf41f0bc6967ce64cc9706: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 04:00:55.234714 14606 tablet_bootstrap.cc:492] T 7aba625221014e3f8069dddce55382d1 P febbc732bc55422e9ee6bdf4c48eec3a: No bootstrap required, opened a new log
I20260926 04:00:55.234789 14606 ts_tablet_manager.cc:1397] T 7aba625221014e3f8069dddce55382d1 P febbc732bc55422e9ee6bdf4c48eec3a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 04:00:55.235158 14608 log.cc:824] T 7aba625221014e3f8069dddce55382d1 P c199a3d5f00d4157bb393acf090b3fda: Log is configured to *not* fsync() on all Append() calls
I20260926 04:00:55.235865 14608 tablet_bootstrap.cc:492] T 7aba625221014e3f8069dddce55382d1 P c199a3d5f00d4157bb393acf090b3fda: No bootstrap required, opened a new log
I20260926 04:00:55.235926 14608 ts_tablet_manager.cc:1397] T 7aba625221014e3f8069dddce55382d1 P c199a3d5f00d4157bb393acf090b3fda: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 04:00:55.236488 14606 raft_consensus.cc:348] T 7aba625221014e3f8069dddce55382d1 P febbc732bc55422e9ee6bdf4c48eec3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9e9f973c5bf41f0bc6967ce64cc9706" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35223 } } peers { permanent_uuid: "febbc732bc55422e9ee6bdf4c48eec3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37355 } } peers { permanent_uuid: "c199a3d5f00d4157bb393acf090b3fda" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37653 } }
I20260926 04:00:55.236610 14606 raft_consensus.cc:374] T 7aba625221014e3f8069dddce55382d1 P febbc732bc55422e9ee6bdf4c48eec3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:00:55.236690 14606 raft_consensus.cc:729] T 7aba625221014e3f8069dddce55382d1 P febbc732bc55422e9ee6bdf4c48eec3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: febbc732bc55422e9ee6bdf4c48eec3a, State: Initialized, Role: FOLLOWER
I20260926 04:00:55.236811 14606 consensus_queue.cc:260] T 7aba625221014e3f8069dddce55382d1 P febbc732bc55422e9ee6bdf4c48eec3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9e9f973c5bf41f0bc6967ce64cc9706" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35223 } } peers { permanent_uuid: "febbc732bc55422e9ee6bdf4c48eec3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37355 } } peers { permanent_uuid: "c199a3d5f00d4157bb393acf090b3fda" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37653 } }
I20260926 04:00:55.236768 14607 raft_consensus.cc:348] T 7aba625221014e3f8069dddce55382d1 P a9e9f973c5bf41f0bc6967ce64cc9706 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9e9f973c5bf41f0bc6967ce64cc9706" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35223 } } peers { permanent_uuid: "febbc732bc55422e9ee6bdf4c48eec3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37355 } } peers { permanent_uuid: "c199a3d5f00d4157bb393acf090b3fda" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37653 } }
I20260926 04:00:55.236948 14607 raft_consensus.cc:374] T 7aba625221014e3f8069dddce55382d1 P a9e9f973c5bf41f0bc6967ce64cc9706 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:00:55.236984 14607 raft_consensus.cc:729] T 7aba625221014e3f8069dddce55382d1 P a9e9f973c5bf41f0bc6967ce64cc9706 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9e9f973c5bf41f0bc6967ce64cc9706, State: Initialized, Role: FOLLOWER
I20260926 04:00:55.237080 14606 ts_tablet_manager.cc:1428] T 7aba625221014e3f8069dddce55382d1 P febbc732bc55422e9ee6bdf4c48eec3a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 04:00:55.237079 14607 consensus_queue.cc:260] T 7aba625221014e3f8069dddce55382d1 P a9e9f973c5bf41f0bc6967ce64cc9706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9e9f973c5bf41f0bc6967ce64cc9706" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35223 } } peers { permanent_uuid: "febbc732bc55422e9ee6bdf4c48eec3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37355 } } peers { permanent_uuid: "c199a3d5f00d4157bb393acf090b3fda" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37653 } }
I20260926 04:00:55.237088 14592 heartbeater.cc:503] Master 127.0.98.62:46215 was elected leader, sending a full tablet report...
I20260926 04:00:55.237352 14607 ts_tablet_manager.cc:1428] T 7aba625221014e3f8069dddce55382d1 P a9e9f973c5bf41f0bc6967ce64cc9706: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 04:00:55.237387 14458 heartbeater.cc:503] Master 127.0.98.62:46215 was elected leader, sending a full tablet report...
I20260926 04:00:55.237910 14608 raft_consensus.cc:348] T 7aba625221014e3f8069dddce55382d1 P c199a3d5f00d4157bb393acf090b3fda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9e9f973c5bf41f0bc6967ce64cc9706" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35223 } } peers { permanent_uuid: "febbc732bc55422e9ee6bdf4c48eec3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37355 } } peers { permanent_uuid: "c199a3d5f00d4157bb393acf090b3fda" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37653 } }
I20260926 04:00:55.238021 14608 raft_consensus.cc:374] T 7aba625221014e3f8069dddce55382d1 P c199a3d5f00d4157bb393acf090b3fda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:00:55.238041 14608 raft_consensus.cc:729] T 7aba625221014e3f8069dddce55382d1 P c199a3d5f00d4157bb393acf090b3fda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c199a3d5f00d4157bb393acf090b3fda, State: Initialized, Role: FOLLOWER
I20260926 04:00:55.238111 14608 consensus_queue.cc:260] T 7aba625221014e3f8069dddce55382d1 P c199a3d5f00d4157bb393acf090b3fda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9e9f973c5bf41f0bc6967ce64cc9706" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35223 } } peers { permanent_uuid: "febbc732bc55422e9ee6bdf4c48eec3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37355 } } peers { permanent_uuid: "c199a3d5f00d4157bb393acf090b3fda" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37653 } }
I20260926 04:00:55.238332 14608 ts_tablet_manager.cc:1428] T 7aba625221014e3f8069dddce55382d1 P c199a3d5f00d4157bb393acf090b3fda: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 04:00:55.238384 14324 heartbeater.cc:503] Master 127.0.98.62:46215 was elected leader, sending a full tablet report...
W20260926 04:00:55.285826 14459 tablet.cc:2369] T 7aba625221014e3f8069dddce55382d1 P a9e9f973c5bf41f0bc6967ce64cc9706: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 04:00:55.388721 14325 tablet.cc:2369] T 7aba625221014e3f8069dddce55382d1 P c199a3d5f00d4157bb393acf090b3fda: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 04:00:55.414162 14612 raft_consensus.cc:482] T 7aba625221014e3f8069dddce55382d1 P febbc732bc55422e9ee6bdf4c48eec3a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 04:00:55.414354 14612 raft_consensus.cc:504] T 7aba625221014e3f8069dddce55382d1 P febbc732bc55422e9ee6bdf4c48eec3a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9e9f973c5bf41f0bc6967ce64cc9706" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35223 } } peers { permanent_uuid: "febbc732bc55422e9ee6bdf4c48eec3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37355 } } peers { permanent_uuid: "c199a3d5f00d4157bb393acf090b3fda" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37653 } }
I20260926 04:00:55.414757 14612 leader_election.cc:288] T 7aba625221014e3f8069dddce55382d1 P febbc732bc55422e9ee6bdf4c48eec3a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a9e9f973c5bf41f0bc6967ce64cc9706 (127.0.98.2:35223), c199a3d5f00d4157bb393acf090b3fda (127.0.98.1:37653)
I20260926 04:00:55.417732 14604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:55.415036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56689 (local address 127.0.98.2:35223)
0926 04:00:55.415082 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 04:00:55.415084 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:00:55.415093 (+ 9us) server_negotiation.cc:407] Connection header received
0926 04:00:55.415140 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:00:55.415141 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:00:55.415154 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:00:55.415229 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:00:55.415382 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:55.415913 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:55.416984 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:55.417370 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:55.417467 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:55.417478 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 04:00:55.417624 (+ 146us) server_negotiation.cc:299] Negotiation successful
0926 04:00:55.417667 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 04:00:55.417759 14594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:55.414963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35223 (local address 127.0.98.3:56689)
0926 04:00:55.415023 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0926 04:00:55.415035 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 04:00:55.415082 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:00:55.415233 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:00:55.415234 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:00:55.415236 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:00:55.415307 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:00:55.415310 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:00:55.415994 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:00:55.415995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:00:55.416892 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:00:55.416895 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:00:55.417467 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:00:55.417468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:00:55.417544 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:55.417547 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 04:00:55.417617 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 04:00:55.417660 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 04:00:55.417797 14605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:55.415023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49371 (local address 127.0.98.1:37653)
0926 04:00:55.415070 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 04:00:55.415072 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:00:55.415219 (+ 147us) server_negotiation.cc:407] Connection header received
0926 04:00:55.415308 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:00:55.415308 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:00:55.415322 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:00:55.415389 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:00:55.415549 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:55.416075 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:55.417072 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:55.417439 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:55.417521 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:55.417531 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 04:00:55.417682 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 04:00:55.417716 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 04:00:55.418030 14615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:55.414985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37653 (local address 127.0.98.3:49371)
0926 04:00:55.415148 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 04:00:55.415156 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 04:00:55.415234 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:00:55.415391 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:00:55.415392 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:00:55.415393 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:00:55.415473 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:00:55.415474 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:00:55.416153 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:00:55.416154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:00:55.416982 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:00:55.416985 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:00:55.417535 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:00:55.417536 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:00:55.417604 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:55.417608 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 04:00:55.417674 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 04:00:55.417713 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":93,"threads_started":1}
I20260926 04:00:55.418102 14412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aba625221014e3f8069dddce55382d1" candidate_uuid: "febbc732bc55422e9ee6bdf4c48eec3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9e9f973c5bf41f0bc6967ce64cc9706" is_pre_election: true
I20260926 04:00:55.418268 14412 raft_consensus.cc:2454] T 7aba625221014e3f8069dddce55382d1 P a9e9f973c5bf41f0bc6967ce64cc9706 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate febbc732bc55422e9ee6bdf4c48eec3a in term 0.
I20260926 04:00:55.418419 14278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aba625221014e3f8069dddce55382d1" candidate_uuid: "febbc732bc55422e9ee6bdf4c48eec3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c199a3d5f00d4157bb393acf090b3fda" is_pre_election: true
I20260926 04:00:55.418529 14479 leader_election.cc:302] T 7aba625221014e3f8069dddce55382d1 P febbc732bc55422e9ee6bdf4c48eec3a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9e9f973c5bf41f0bc6967ce64cc9706, febbc732bc55422e9ee6bdf4c48eec3a; no voters:
I20260926 04:00:55.418605 14278 raft_consensus.cc:2454] T 7aba625221014e3f8069dddce55382d1 P c199a3d5f00d4157bb393acf090b3fda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate febbc732bc55422e9ee6bdf4c48eec3a in term 0.
I20260926 04:00:55.418684 14612 raft_consensus.cc:2781] T 7aba625221014e3f8069dddce55382d1 P febbc732bc55422e9ee6bdf4c48eec3a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 04:00:55.418741 14612 raft_consensus.cc:482] T 7aba625221014e3f8069dddce55382d1 P febbc732bc55422e9ee6bdf4c48eec3a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 04:00:55.418766 14612 raft_consensus.cc:3037] T 7aba625221014e3f8069dddce55382d1 P febbc732bc55422e9ee6bdf4c48eec3a [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:00:55.419468 14612 raft_consensus.cc:504] T 7aba625221014e3f8069dddce55382d1 P febbc732bc55422e9ee6bdf4c48eec3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9e9f973c5bf41f0bc6967ce64cc9706" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35223 } } peers { permanent_uuid: "febbc732bc55422e9ee6bdf4c48eec3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37355 } } peers { permanent_uuid: "c199a3d5f00d4157bb393acf090b3fda" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37653 } }
I20260926 04:00:55.419602 14612 leader_election.cc:288] T 7aba625221014e3f8069dddce55382d1 P febbc732bc55422e9ee6bdf4c48eec3a [CANDIDATE]: Term 1 election: Requested vote from peers a9e9f973c5bf41f0bc6967ce64cc9706 (127.0.98.2:35223), c199a3d5f00d4157bb393acf090b3fda (127.0.98.1:37653)
I20260926 04:00:55.419788 14412 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aba625221014e3f8069dddce55382d1" candidate_uuid: "febbc732bc55422e9ee6bdf4c48eec3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9e9f973c5bf41f0bc6967ce64cc9706"
I20260926 04:00:55.419814 14278 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7aba625221014e3f8069dddce55382d1" candidate_uuid: "febbc732bc55422e9ee6bdf4c48eec3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c199a3d5f00d4157bb393acf090b3fda"
I20260926 04:00:55.419874 14278 raft_consensus.cc:3037] T 7aba625221014e3f8069dddce55382d1 P c199a3d5f00d4157bb393acf090b3fda [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:00:55.419893 14412 raft_consensus.cc:3037] T 7aba625221014e3f8069dddce55382d1 P a9e9f973c5bf41f0bc6967ce64cc9706 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:00:55.420596 14412 raft_consensus.cc:2454] T 7aba625221014e3f8069dddce55382d1 P a9e9f973c5bf41f0bc6967ce64cc9706 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate febbc732bc55422e9ee6bdf4c48eec3a in term 1.
I20260926 04:00:55.420625 14278 raft_consensus.cc:2454] T 7aba625221014e3f8069dddce55382d1 P c199a3d5f00d4157bb393acf090b3fda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate febbc732bc55422e9ee6bdf4c48eec3a in term 1.
I20260926 04:00:55.420787 14479 leader_election.cc:302] T 7aba625221014e3f8069dddce55382d1 P febbc732bc55422e9ee6bdf4c48eec3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9e9f973c5bf41f0bc6967ce64cc9706, febbc732bc55422e9ee6bdf4c48eec3a; no voters:
I20260926 04:00:55.420874 14612 raft_consensus.cc:2781] T 7aba625221014e3f8069dddce55382d1 P febbc732bc55422e9ee6bdf4c48eec3a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:00:55.421000 14612 raft_consensus.cc:686] T 7aba625221014e3f8069dddce55382d1 P febbc732bc55422e9ee6bdf4c48eec3a [term 1 LEADER]: Becoming Leader. State: Replica: febbc732bc55422e9ee6bdf4c48eec3a, State: Running, Role: LEADER
I20260926 04:00:55.421064 14612 consensus_queue.cc:237] T 7aba625221014e3f8069dddce55382d1 P febbc732bc55422e9ee6bdf4c48eec3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9e9f973c5bf41f0bc6967ce64cc9706" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35223 } } peers { permanent_uuid: "febbc732bc55422e9ee6bdf4c48eec3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37355 } } peers { permanent_uuid: "c199a3d5f00d4157bb393acf090b3fda" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37653 } }
I20260926 04:00:55.421914 14133 catalog_manager.cc:5495] T 7aba625221014e3f8069dddce55382d1 P febbc732bc55422e9ee6bdf4c48eec3a reported cstate change: term changed from 0 to 1, leader changed from <none> to febbc732bc55422e9ee6bdf4c48eec3a (127.0.98.3). New cstate: current_term: 1 leader_uuid: "febbc732bc55422e9ee6bdf4c48eec3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9e9f973c5bf41f0bc6967ce64cc9706" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "febbc732bc55422e9ee6bdf4c48eec3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c199a3d5f00d4157bb393acf090b3fda" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37653 } health_report { overall_health: UNKNOWN } } }
W20260926 04:00:55.446359 14593 tablet.cc:2369] T 7aba625221014e3f8069dddce55382d1 P febbc732bc55422e9ee6bdf4c48eec3a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 04:00:55.452883 14603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:55.450616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45846 (local address 127.0.98.3:37355)
0926 04:00:55.450686 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 04:00:55.450688 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:00:55.450714 (+ 26us) server_negotiation.cc:407] Connection header received
0926 04:00:55.450747 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:00:55.450748 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:00:55.450765 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:00:55.450817 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 04:00:55.450947 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:55.451407 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:55.452135 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:55.452312 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:55.452364 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:55.452490 (+ 126us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 04:00:55.452583 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 04:00:55.452690 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 04:00:55.452733 (+ 43us) server_negotiation.cc:299] Negotiation successful
0926 04:00:55.452757 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 04:00:55.454727 14278 raft_consensus.cc:1261] T 7aba625221014e3f8069dddce55382d1 P c199a3d5f00d4157bb393acf090b3fda [term 1 FOLLOWER]: Refusing update from remote peer febbc732bc55422e9ee6bdf4c48eec3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:00:55.454742 14412 raft_consensus.cc:1261] T 7aba625221014e3f8069dddce55382d1 P a9e9f973c5bf41f0bc6967ce64cc9706 [term 1 FOLLOWER]: Refusing update from remote peer febbc732bc55422e9ee6bdf4c48eec3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:00:55.455089 14616 consensus_queue.cc:1035] T 7aba625221014e3f8069dddce55382d1 P febbc732bc55422e9ee6bdf4c48eec3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c199a3d5f00d4157bb393acf090b3fda" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:00:55.455231 14612 consensus_queue.cc:1035] T 7aba625221014e3f8069dddce55382d1 P febbc732bc55422e9ee6bdf4c48eec3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9e9f973c5bf41f0bc6967ce64cc9706" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:00:55.457734 14621 mvcc.cc:205] Tried to move back new op lower bound from 7333458966339477504 to 7333458966205272064. Current Snapshot: MvccSnapshot[applied={T|T < 7333458966339477504}]
I20260926 04:00:55.458151 14620 mvcc.cc:205] Tried to move back new op lower bound from 7333458966339477504 to 7333458966205272064. Current Snapshot: MvccSnapshot[applied={T|T < 7333458966339477504}]
I20260926 04:00:55.459679 14132 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45912:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 04:00:55.459776 14132 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45912:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 04:00:55.460561 14132 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a2e82428eba746a8b01d279a4f8b659f: Sending DeleteTablet for 3 replicas of tablet 7aba625221014e3f8069dddce55382d1
I20260926 04:00:55.460887 14526 tablet_service.cc:1514] Processing DeleteTablet for tablet 7aba625221014e3f8069dddce55382d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:00:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45832
I20260926 04:00:55.460929 14392 tablet_service.cc:1514] Processing DeleteTablet for tablet 7aba625221014e3f8069dddce55382d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:00:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54862
I20260926 04:00:55.460937 14258 tablet_service.cc:1514] Processing DeleteTablet for tablet 7aba625221014e3f8069dddce55382d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:00:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60648
I20260926 04:00:55.461102 14629 tablet_replica.cc:330] T 7aba625221014e3f8069dddce55382d1 P febbc732bc55422e9ee6bdf4c48eec3a: stopping tablet replica
I20260926 04:00:55.461212 14631 tablet_replica.cc:330] T 7aba625221014e3f8069dddce55382d1 P c199a3d5f00d4157bb393acf090b3fda: stopping tablet replica
I20260926 04:00:55.461212 14629 raft_consensus.cc:2229] T 7aba625221014e3f8069dddce55382d1 P febbc732bc55422e9ee6bdf4c48eec3a [term 1 LEADER]: Raft consensus shutting down.
I20260926 04:00:55.461215 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 14195
I20260926 04:00:55.461323 14629 raft_consensus.cc:2258] T 7aba625221014e3f8069dddce55382d1 P febbc732bc55422e9ee6bdf4c48eec3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:00:55.463194 14629 ts_tablet_manager.cc:1905] T 7aba625221014e3f8069dddce55382d1 P febbc732bc55422e9ee6bdf4c48eec3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:00:55.464757 14630 tablet_replica.cc:330] T 7aba625221014e3f8069dddce55382d1 P a9e9f973c5bf41f0bc6967ce64cc9706: stopping tablet replica
I20260926 04:00:55.464916 14630 raft_consensus.cc:2229] T 7aba625221014e3f8069dddce55382d1 P a9e9f973c5bf41f0bc6967ce64cc9706 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 04:00:55.465085 14630 raft_consensus.cc:2258] T 7aba625221014e3f8069dddce55382d1 P a9e9f973c5bf41f0bc6967ce64cc9706 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:00:55.465256 14629 ts_tablet_manager.cc:1918] T 7aba625221014e3f8069dddce55382d1 P febbc732bc55422e9ee6bdf4c48eec3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 04:00:55.465356 14629 log.cc:1196] T 7aba625221014e3f8069dddce55382d1 P febbc732bc55422e9ee6bdf4c48eec3a: Deleting WAL directory at /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-2/wal/wals/7aba625221014e3f8069dddce55382d1
I20260926 04:00:55.465658 14629 ts_tablet_manager.cc:1939] T 7aba625221014e3f8069dddce55382d1 P febbc732bc55422e9ee6bdf4c48eec3a: Deleting consensus metadata
I20260926 04:00:55.466048 14119 catalog_manager.cc:4841] TS febbc732bc55422e9ee6bdf4c48eec3a (127.0.98.3:37355): tablet 7aba625221014e3f8069dddce55382d1 (table test-table [id=cf25aa26c03f48a19dbc9cb2bd83c91d]) successfully deleted
I20260926 04:00:55.466099 14630 ts_tablet_manager.cc:1905] T 7aba625221014e3f8069dddce55382d1 P a9e9f973c5bf41f0bc6967ce64cc9706: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 04:00:55.467402 14118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37653 (error 108)
I20260926 04:00:55.467658 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 14329
I20260926 04:00:55.472009 14600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:55.471879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37653 (local address 127.0.0.1:60658)
0926 04:00:55.471935 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 04:00:55.471974 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 04:00:55.472194 14118 catalog_manager.cc:4584] TS c199a3d5f00d4157bb393acf090b3fda (127.0.98.1:37653): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7aba625221014e3f8069dddce55382d1: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37653: connect: Connection refused (error 111)
I20260926 04:00:55.474468 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 14463
I20260926 04:00:55.474499 14600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:55.474409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35223 (local address 127.0.0.1:54876)
0926 04:00:55.474450 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0926 04:00:55.474472 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:35223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260926 04:00:55.479005 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 14101
I20260926 04:00:55.483871 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46215
--webserver_interface=127.0.98.62
--webserver_port=34903
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46215
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:00:55.566191 14633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:00:55.566397 14633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:00:55.566419 14633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:00:55.567586 14633 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:00:55.567628 14633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:00:55.567642 14633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:00:55.567684 14633 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:00:55.567714 14633 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:00:55.569198 14633 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46215
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=34903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14633
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:00:55.569450 14633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:00:55.569710 14633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:00:55.578291 14642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:00:55.578437 14633 server_base.cc:1034] running on GCE node
W20260926 04:00:55.578291 14640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:00:55.578323 14639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:00:55.578868 14633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:00:55.579206 14633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:00:55.580363 14633 hybrid_clock.cc:648] HybridClock initialized: now 1790395255580356 us; error 31 us; skew 500 ppm
Sep 26 04:00:55 dist-test-slave-nhpw krb5kdc[14088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395255, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:00:55.583014 14633 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 04:00:55.583467 14633 webserver.cc:460] Webserver started at http://127.0.98.62:34903/ using document root <none> and password file <none>
I20260926 04:00:55.583714 14633 fs_manager.cc:359] Metadata directory not provided
I20260926 04:00:55.583796 14633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:00:55.585242 14633 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 04:00:55.585866 14649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:00:55.586035 14633 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 04:00:55.586143 14633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/master-0/wal
uuid: "a2e82428eba746a8b01d279a4f8b659f"
format_stamp: "Formatted at 2026-09-26 04:00:54 on dist-test-slave-nhpw"
server_key: "88b3ac9de17d77ea99a6433a2d34f8e7"
server_key_iv: "b1352df44537b6a5c96335f7bf538c1c"
server_key_version: "encryptionkey@0"
I20260926 04:00:55.586560 14633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:00:55.597383 14633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:00:55.599641 14633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:00:55.599807 14633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:00:55.603912 14633 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46215
I20260926 04:00:55.603955 14701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46215 every 8 connection(s)
I20260926 04:00:55.604344 14633 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
I20260926 04:00:55.605952 14702 sys_catalog.cc:263] Verifying existing consensus state
I20260926 04:00:55.606622 14702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2e82428eba746a8b01d279a4f8b659f: Bootstrap starting.
I20260926 04:00:55.608475 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 14633
I20260926 04:00:55.608803 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:37653
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:46215
--webserver_port=34091
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 04:00:55.615373 14702 log.cc:824] T 00000000000000000000000000000000 P a2e82428eba746a8b01d279a4f8b659f: Log is configured to *not* fsync() on all Append() calls
I20260926 04:00:55.617416 14702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2e82428eba746a8b01d279a4f8b659f: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 04:00:55.617699 14702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2e82428eba746a8b01d279a4f8b659f: Bootstrap complete.
I20260926 04:00:55.619944 14702 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2e82428eba746a8b01d279a4f8b659f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2e82428eba746a8b01d279a4f8b659f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46215 } }
I20260926 04:00:55.620193 14702 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2e82428eba746a8b01d279a4f8b659f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e82428eba746a8b01d279a4f8b659f, State: Initialized, Role: FOLLOWER
I20260926 04:00:55.620398 14702 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2e82428eba746a8b01d279a4f8b659f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2e82428eba746a8b01d279a4f8b659f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46215 } }
I20260926 04:00:55.620494 14702 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2e82428eba746a8b01d279a4f8b659f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:00:55.620548 14702 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2e82428eba746a8b01d279a4f8b659f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:00:55.620594 14702 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2e82428eba746a8b01d279a4f8b659f [term 1 FOLLOWER]: Advancing to term 2
I20260926 04:00:55.621625 14702 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2e82428eba746a8b01d279a4f8b659f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2e82428eba746a8b01d279a4f8b659f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46215 } }
I20260926 04:00:55.621744 14702 leader_election.cc:302] T 00000000000000000000000000000000 P a2e82428eba746a8b01d279a4f8b659f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2e82428eba746a8b01d279a4f8b659f; no voters:
I20260926 04:00:55.621928 14702 leader_election.cc:288] T 00000000000000000000000000000000 P a2e82428eba746a8b01d279a4f8b659f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 04:00:55.622046 14706 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2e82428eba746a8b01d279a4f8b659f [term 2 FOLLOWER]: Leader election won for term 2
I20260926 04:00:55.622188 14702 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2e82428eba746a8b01d279a4f8b659f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:00:55.622217 14706 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2e82428eba746a8b01d279a4f8b659f [term 2 LEADER]: Becoming Leader. State: Replica: a2e82428eba746a8b01d279a4f8b659f, State: Running, Role: LEADER
I20260926 04:00:55.622301 14706 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2e82428eba746a8b01d279a4f8b659f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2e82428eba746a8b01d279a4f8b659f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46215 } }
I20260926 04:00:55.622611 14706 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2e82428eba746a8b01d279a4f8b659f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2e82428eba746a8b01d279a4f8b659f. Latest consensus state: current_term: 2 leader_uuid: "a2e82428eba746a8b01d279a4f8b659f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2e82428eba746a8b01d279a4f8b659f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46215 } } }
I20260926 04:00:55.622728 14706 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2e82428eba746a8b01d279a4f8b659f [sys.catalog]: This master's current role is: LEADER
I20260926 04:00:55.622612 14707 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2e82428eba746a8b01d279a4f8b659f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a2e82428eba746a8b01d279a4f8b659f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2e82428eba746a8b01d279a4f8b659f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46215 } } }
I20260926 04:00:55.622912 14707 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2e82428eba746a8b01d279a4f8b659f [sys.catalog]: This master's current role is: LEADER
I20260926 04:00:55.623258 14713 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:00:55.624063 14713 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:00:55.624182 14713 catalog_manager.cc:1259] Loaded cluster ID: 8d7e2452f04944cb98926530d868de23
I20260926 04:00:55.624230 14713 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 04:00:55.624625 14713 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260926 04:00:55.629690 14713 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260926 04:00:55.632083 14713 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:00:55.632304 14713 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a2e82428eba746a8b01d279a4f8b659f: Loaded TSK: 0
I20260926 04:00:55.632527 14713 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 04:00:55.693033 14704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:00:55.693236 14704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:00:55.693259 14704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:00:55.694396 14704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:00:55.694435 14704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:00:55.694461 14704 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 04:00:55.695876 14704 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:37653
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=34091
--tserver_master_addrs=127.0.98.62:46215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:00:55.696105 14704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:00:55.696347 14704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:00:55.701251 14729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:00:55.701295 14732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:00:55.701428 14704 server_base.cc:1034] running on GCE node
W20260926 04:00:55.701524 14730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:00:55.701704 14704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:00:55.701942 14704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:00:55.703104 14704 hybrid_clock.cc:648] HybridClock initialized: now 1790395255703090 us; error 30 us; skew 500 ppm
Sep 26 04:00:55 dist-test-slave-nhpw krb5kdc[14088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395255, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:00:55.705637 14704 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 04:00:55.706097 14704 webserver.cc:460] Webserver started at http://127.0.98.1:34091/ using document root <none> and password file <none>
I20260926 04:00:55.706334 14704 fs_manager.cc:359] Metadata directory not provided
I20260926 04:00:55.706413 14704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:00:55.707798 14704 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:00:55.708485 14739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:00:55.708678 14704 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:00:55.708784 14704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-0/wal
uuid: "c199a3d5f00d4157bb393acf090b3fda"
format_stamp: "Formatted at 2026-09-26 04:00:54 on dist-test-slave-nhpw"
server_key: "13f7f460d5365dcbd0448ac676ca5d87"
server_key_iv: "75b9a57771d10f436ee080c912d88320"
server_key_version: "encryptionkey@0"
I20260926 04:00:55.709110 14704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:00:55.738427 14704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:00:55.740885 14704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:00:55.741037 14704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:00:55.741291 14704 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:00:55.741797 14746 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 04:00:55.743219 14704 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 04:00:55.743270 14704 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 04:00:55.743324 14704 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 04:00:55.743959 14704 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 04:00:55.744000 14704 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 04:00:55.744069 14746 tablet_bootstrap.cc:492] T 7aba625221014e3f8069dddce55382d1 P c199a3d5f00d4157bb393acf090b3fda: Bootstrap starting.
I20260926 04:00:55.751710 14704 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37653
I20260926 04:00:55.751827 14853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37653 every 8 connection(s)
I20260926 04:00:55.752252 14704 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
I20260926 04:00:55.754388 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 14704
I20260926 04:00:55.754698 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:35223
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:46215
--webserver_port=40201
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 04:00:55.755188 14746 log.cc:824] T 7aba625221014e3f8069dddce55382d1 P c199a3d5f00d4157bb393acf090b3fda: Log is configured to *not* fsync() on all Append() calls
I20260926 04:00:55.757903 14746 tablet_bootstrap.cc:492] T 7aba625221014e3f8069dddce55382d1 P c199a3d5f00d4157bb393acf090b3fda: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 04:00:55.758180 14746 tablet_bootstrap.cc:492] T 7aba625221014e3f8069dddce55382d1 P c199a3d5f00d4157bb393acf090b3fda: Bootstrap complete.
I20260926 04:00:55.759361 14746 ts_tablet_manager.cc:1397] T 7aba625221014e3f8069dddce55382d1 P c199a3d5f00d4157bb393acf090b3fda: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.004s
Sep 26 04:00:55 dist-test-slave-nhpw krb5kdc[14088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395255, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 04:00:55.760447 14746 raft_consensus.cc:348] T 7aba625221014e3f8069dddce55382d1 P c199a3d5f00d4157bb393acf090b3fda [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9e9f973c5bf41f0bc6967ce64cc9706" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35223 } } peers { permanent_uuid: "febbc732bc55422e9ee6bdf4c48eec3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37355 } } peers { permanent_uuid: "c199a3d5f00d4157bb393acf090b3fda" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37653 } }
I20260926 04:00:55.760752 14746 raft_consensus.cc:729] T 7aba625221014e3f8069dddce55382d1 P c199a3d5f00d4157bb393acf090b3fda [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c199a3d5f00d4157bb393acf090b3fda, State: Initialized, Role: FOLLOWER
I20260926 04:00:55.760901 14746 consensus_queue.cc:260] T 7aba625221014e3f8069dddce55382d1 P c199a3d5f00d4157bb393acf090b3fda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9e9f973c5bf41f0bc6967ce64cc9706" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35223 } } peers { permanent_uuid: "febbc732bc55422e9ee6bdf4c48eec3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37355 } } peers { permanent_uuid: "c199a3d5f00d4157bb393acf090b3fda" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37653 } }
I20260926 04:00:55.761164 14746 ts_tablet_manager.cc:1428] T 7aba625221014e3f8069dddce55382d1 P c199a3d5f00d4157bb393acf090b3fda: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 04:00:55.765125 14856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:55.753003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.98.1:37051)
0926 04:00:55.753210 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0926 04:00:55.753254 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 04:00:55.753763 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:00:55.754238 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:00:55.754242 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:00:55.754592 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:00:55.754855 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:00:55.754863 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:00:55.756389 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:00:55.756391 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:00:55.757675 (+ 1284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:00:55.757678 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:00:55.758418 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:00:55.758419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:00:55.758571 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:55.759203 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:00:55.759213 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:00:55.761094 (+ 1881us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:00:55.763471 (+ 2377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:00:55.763478 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:00:55.763480 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:00:55.763698 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:00:55.763999 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:00:55.763999 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:00:55.764000 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:00:55.764106 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:00:55.764580 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:00:55.764585 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:00:55.764770 (+ 185us) client_negotiation.cc:769] Sending connection context
0926 04:00:55.764860 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 04:00:55.764984 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":124,"threads_started":1}
I20260926 04:00:55.765343 14857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:55.753115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37051 (local address 127.0.98.62:46215)
0926 04:00:55.753361 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0926 04:00:55.753369 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 04:00:55.753406 (+ 37us) server_negotiation.cc:407] Connection header received
0926 04:00:55.754001 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:00:55.754006 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:00:55.754055 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:00:55.754196 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:00:55.754988 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:55.756258 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:55.757778 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:55.758311 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:55.758418 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:55.761238 (+ 2820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:00:55.761243 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:00:55.761244 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:00:55.761258 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:00:55.763376 (+ 2118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:00:55.763792 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:00:55.763795 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:00:55.763796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:00:55.763924 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:00:55.764177 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:00:55.764178 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:00:55.764178 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:00:55.764501 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:00:55.764600 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 04:00:55.764945 (+ 345us) server_negotiation.cc:299] Negotiation successful
0926 04:00:55.765107 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":99,"threads_started":1}
I20260926 04:00:55.765988 14854 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46215
I20260926 04:00:55.766100 14854 heartbeater.cc:464] Registering TS with master...
I20260926 04:00:55.766287 14854 heartbeater.cc:511] Master 127.0.98.62:46215 requested a full tablet report, sending...
I20260926 04:00:55.766997 14666 ts_manager.cc:194] Registered new tserver with Master: c199a3d5f00d4157bb393acf090b3fda (127.0.98.1:37653)
I20260926 04:00:55.768085 14666 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37051
I20260926 04:00:55.769037 14854 heartbeater.cc:503] Master 127.0.98.62:46215 was elected leader, sending a full tablet report...
I20260926 04:00:55.772384 14861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:55.767569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37653 (local address 127.0.0.1:60674)
0926 04:00:55.767779 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 04:00:55.767807 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 04:00:55.767863 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:00:55.769061 (+ 1198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:00:55.769064 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:00:55.769066 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:00:55.769152 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:00:55.769154 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:00:55.770512 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:00:55.770513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:00:55.771307 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:00:55.771310 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:00:55.772172 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:00:55.772173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:00:55.772229 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:55.772238 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 04:00:55.772292 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 04:00:55.772331 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":95,"threads_started":1}
I20260926 04:00:55.772426 14862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:55.767636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60674 (local address 127.0.98.1:37653)
0926 04:00:55.768744 (+ 1108us) server_negotiation.cc:206] Beginning negotiation
0926 04:00:55.768750 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 04:00:55.768763 (+ 13us) server_negotiation.cc:407] Connection header received
0926 04:00:55.768789 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:00:55.768790 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:00:55.768938 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:00:55.769053 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:00:55.769298 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:55.770428 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:55.771389 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:55.772092 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:55.772166 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:55.772217 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 04:00:55.772320 (+ 103us) server_negotiation.cc:299] Negotiation successful
0926 04:00:55.772357 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1079,"thread_start_us":69,"threads_started":1}
I20260926 04:00:55.772817 14788 tablet_service.cc:1514] Processing DeleteTablet for tablet 7aba625221014e3f8069dddce55382d1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:00:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60674
I20260926 04:00:55.773439 14863 tablet_replica.cc:330] stopping tablet replica
I20260926 04:00:55.773556 14863 raft_consensus.cc:2229] T 7aba625221014e3f8069dddce55382d1 P c199a3d5f00d4157bb393acf090b3fda [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 04:00:55.773645 14863 raft_consensus.cc:2258] T 7aba625221014e3f8069dddce55382d1 P c199a3d5f00d4157bb393acf090b3fda [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:00:55.773941 14863 ts_tablet_manager.cc:1905] T 7aba625221014e3f8069dddce55382d1 P c199a3d5f00d4157bb393acf090b3fda: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:00:55.776041 14863 ts_tablet_manager.cc:1918] T 7aba625221014e3f8069dddce55382d1 P c199a3d5f00d4157bb393acf090b3fda: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 04:00:55.776134 14863 log.cc:1196] T 7aba625221014e3f8069dddce55382d1 P c199a3d5f00d4157bb393acf090b3fda: Deleting WAL directory at /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-0/wal/wals/7aba625221014e3f8069dddce55382d1
I20260926 04:00:55.776358 14863 ts_tablet_manager.cc:1939] T 7aba625221014e3f8069dddce55382d1 P c199a3d5f00d4157bb393acf090b3fda: Deleting consensus metadata
I20260926 04:00:55.776886 14651 catalog_manager.cc:4841] TS c199a3d5f00d4157bb393acf090b3fda (127.0.98.1:37653): tablet 7aba625221014e3f8069dddce55382d1 (table test-table [id=cf25aa26c03f48a19dbc9cb2bd83c91d]) successfully deleted
W20260926 04:00:55.841745 14859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:00:55.842020 14859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:00:55.842061 14859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:00:55.843328 14859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:00:55.843410 14859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:00:55.843459 14859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 04:00:55.845028 14859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:35223
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=40201
--tserver_master_addrs=127.0.98.62:46215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:00:55.845316 14859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:00:55.845566 14859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:00:55.850665 14871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:00:55.850713 14868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:00:55.850811 14869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:00:55.850931 14859 server_base.cc:1034] running on GCE node
I20260926 04:00:55.851141 14859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:00:55.851420 14859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:00:55.852588 14859 hybrid_clock.cc:648] HybridClock initialized: now 1790395255852569 us; error 38 us; skew 500 ppm
Sep 26 04:00:55 dist-test-slave-nhpw krb5kdc[14088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395255, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:00:55.855037 14859 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 04:00:55.855513 14859 webserver.cc:460] Webserver started at http://127.0.98.2:40201/ using document root <none> and password file <none>
I20260926 04:00:55.855698 14859 fs_manager.cc:359] Metadata directory not provided
I20260926 04:00:55.855749 14859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:00:55.857239 14859 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 04:00:55.857915 14878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:00:55.858115 14859 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 04:00:55.858201 14859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-1/wal
uuid: "a9e9f973c5bf41f0bc6967ce64cc9706"
format_stamp: "Formatted at 2026-09-26 04:00:55 on dist-test-slave-nhpw"
server_key: "0bdff880fe199481e5f395441305c8ea"
server_key_iv: "5ec0f94efb527d4d81c2fc88a306ae2c"
server_key_version: "encryptionkey@0"
I20260926 04:00:55.858536 14859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:00:55.881049 14859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:00:55.883512 14859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:00:55.883656 14859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:00:55.883942 14859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:00:55.884389 14885 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260926 04:00:55.885900 14885 ts_tablet_manager.cc:1852] T 7aba625221014e3f8069dddce55382d1 P a9e9f973c5bf41f0bc6967ce64cc9706: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260926 04:00:55.886170 14885 ts_tablet_manager.cc:1905] T 7aba625221014e3f8069dddce55382d1 P a9e9f973c5bf41f0bc6967ce64cc9706: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:00:55.888151 14885 ts_tablet_manager.cc:1918] T 7aba625221014e3f8069dddce55382d1 P a9e9f973c5bf41f0bc6967ce64cc9706: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 04:00:55.888233 14885 log.cc:1196] T 7aba625221014e3f8069dddce55382d1 P a9e9f973c5bf41f0bc6967ce64cc9706: Deleting WAL directory at /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-1/wal/wals/7aba625221014e3f8069dddce55382d1
I20260926 04:00:55.888432 14885 ts_tablet_manager.cc:1939] T 7aba625221014e3f8069dddce55382d1 P a9e9f973c5bf41f0bc6967ce64cc9706: Deleting consensus metadata
I20260926 04:00:55.888602 14859 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260926 04:00:55.888715 14859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260926 04:00:55.888761 14859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:00:55.888792 14859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:00:55.894554 14859 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35223
I20260926 04:00:55.894644 14992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35223 every 8 connection(s)
I20260926 04:00:55.894948 14859 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
Sep 26 04:00:55 dist-test-slave-nhpw krb5kdc[14088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395255, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 04:00:55.901713 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 14859
I20260926 04:00:55.902005 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:37355
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:46215
--webserver_port=42519
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 04:00:55.904780 14857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:55.895756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47193 (local address 127.0.98.62:46215)
0926 04:00:55.895827 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 04:00:55.895830 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:00:55.896068 (+ 238us) server_negotiation.cc:407] Connection header received
0926 04:00:55.896559 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:00:55.896560 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:00:55.896571 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:00:55.896720 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:00:55.897329 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:55.897901 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:55.898873 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:55.899083 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:55.899146 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:55.901252 (+ 2106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:00:55.901254 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:00:55.901254 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:00:55.901261 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:00:55.903022 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:00:55.903501 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:00:55.903503 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:00:55.903504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:00:55.903574 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:00:55.903960 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:00:55.903962 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:00:55.903962 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:00:55.904141 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:00:55.904242 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 04:00:55.904540 (+ 298us) server_negotiation.cc:299] Negotiation successful
0926 04:00:55.904671 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 04:00:55.905014 14995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:55.895759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.98.2:47193)
0926 04:00:55.895950 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 04:00:55.895986 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 04:00:55.896450 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:00:55.896774 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:00:55.896778 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:00:55.897029 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:00:55.897233 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:00:55.897237 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:00:55.897987 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:00:55.897988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:00:55.898783 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:00:55.898785 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:00:55.899161 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:00:55.899162 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:00:55.899261 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:55.899758 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:00:55.899766 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:00:55.901141 (+ 1375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:00:55.903121 (+ 1980us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:00:55.903127 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:00:55.903129 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:00:55.903379 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:00:55.903662 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:00:55.903663 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:00:55.903663 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:00:55.903843 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:00:55.904232 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:00:55.904239 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:00:55.904392 (+ 153us) client_negotiation.cc:769] Sending connection context
0926 04:00:55.904515 (+ 123us) client_negotiation.cc:240] Negotiation successful
0926 04:00:55.904747 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":106,"threads_started":1}
I20260926 04:00:55.905575 14993 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46215
I20260926 04:00:55.905706 14993 heartbeater.cc:464] Registering TS with master...
I20260926 04:00:55.905895 14993 heartbeater.cc:511] Master 127.0.98.62:46215 requested a full tablet report, sending...
I20260926 04:00:55.906363 14666 ts_manager.cc:194] Registered new tserver with Master: a9e9f973c5bf41f0bc6967ce64cc9706 (127.0.98.2:35223)
I20260926 04:00:55.906929 14666 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47193
W20260926 04:00:55.983906 14996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:00:55.984123 14996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:00:55.984145 14996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:00:55.985299 14996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:00:55.985342 14996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:00:55.985369 14996 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 04:00:55.986739 14996 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:37355
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=42519
--tserver_master_addrs=127.0.98.62:46215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:00:55.986986 14996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:00:55.987188 14996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:00:55.992422 15002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:00:55.992429 15001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:00:55.992596 14996 server_base.cc:1034] running on GCE node
W20260926 04:00:55.992429 15004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:00:55.992967 14996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:00:55.993225 14996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:00:55.994390 14996 hybrid_clock.cc:648] HybridClock initialized: now 1790395255994392 us; error 38 us; skew 500 ppm
Sep 26 04:00:55 dist-test-slave-nhpw krb5kdc[14088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395255, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:00:56.000995 14996 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 04:00:56.001489 14996 webserver.cc:460] Webserver started at http://127.0.98.3:42519/ using document root <none> and password file <none>
I20260926 04:00:56.001727 14996 fs_manager.cc:359] Metadata directory not provided
I20260926 04:00:56.001816 14996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:00:56.003209 14996 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 04:00:56.003824 15011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:00:56.004040 14996 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:00:56.004132 14996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-2/wal
uuid: "febbc732bc55422e9ee6bdf4c48eec3a"
format_stamp: "Formatted at 2026-09-26 04:00:55 on dist-test-slave-nhpw"
server_key: "1c1cff0e74cb94c8d1182ad91ac5801f"
server_key_iv: "35d44f9bc936126682151ef75e1c830d"
server_key_version: "encryptionkey@0"
I20260926 04:00:56.004458 14996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:00:56.045198 14996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:00:56.047322 14996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:00:56.047457 14996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:00:56.047703 14996 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:00:56.048002 14996 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:00:56.048034 14996 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:00:56.048058 14996 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:00:56.048071 14996 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:00:56.054003 14996 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37355
I20260926 04:00:56.054049 15124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37355 every 8 connection(s)
I20260926 04:00:56.054401 14996 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
I20260926 04:00:56.057927 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 14996
Sep 26 04:00:56 dist-test-slave-nhpw krb5kdc[14088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395255, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 04:00:56.065174 14857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:56.055201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40949 (local address 127.0.98.62:46215)
0926 04:00:56.055297 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0926 04:00:56.055300 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:00:56.055391 (+ 91us) server_negotiation.cc:407] Connection header received
0926 04:00:56.055905 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:00:56.055906 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:00:56.055916 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:00:56.055970 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:00:56.057103 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:56.057606 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:56.058959 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:56.059270 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:56.059333 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:56.061838 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:00:56.061839 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:00:56.061839 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:00:56.061846 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:00:56.063609 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:00:56.064060 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:00:56.064061 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:00:56.064061 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:00:56.064122 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:00:56.064361 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:00:56.064362 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:00:56.064362 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:00:56.064503 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:00:56.064568 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 04:00:56.064937 (+ 369us) server_negotiation.cc:299] Negotiation successful
0926 04:00:56.065028 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 04:00:56.065297 15129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:56.058349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45914 (local address 127.0.98.62:46215)
0926 04:00:56.058504 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 04:00:56.058505 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 04:00:56.058515 (+ 10us) server_negotiation.cc:407] Connection header received
0926 04:00:56.058543 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:00:56.058543 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:00:56.058554 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:00:56.058626 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:00:56.059224 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:56.059885 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:56.060685 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:56.060881 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:56.060935 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:56.062285 (+ 1350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:00:56.062287 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:00:56.062287 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:00:56.062296 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:00:56.064037 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:00:56.064403 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:00:56.064404 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:00:56.064404 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:00:56.064452 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:00:56.064679 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:00:56.064680 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:00:56.064681 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:00:56.064819 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:00:56.065013 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0926 04:00:56.065073 (+ 60us) server_negotiation.cc:299] Negotiation successful
0926 04:00:56.065167 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":89,"threads_started":1}
I20260926 04:00:56.065354 15127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:56.055066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.98.3:40949)
0926 04:00:56.055239 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 04:00:56.055296 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 04:00:56.055816 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:00:56.056029 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:00:56.056032 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:00:56.056755 (+ 723us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:00:56.056970 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:00:56.056974 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:00:56.057707 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:00:56.057708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:00:56.058877 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:00:56.058879 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:00:56.059359 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:00:56.059360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:00:56.059491 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:56.060053 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:00:56.060061 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:00:56.061739 (+ 1678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:00:56.063734 (+ 1995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:00:56.063740 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:00:56.063742 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:00:56.063971 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:00:56.064181 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:00:56.064182 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:00:56.064182 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:00:56.064292 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:00:56.064557 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:00:56.064561 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:00:56.064785 (+ 224us) client_negotiation.cc:769] Sending connection context
0926 04:00:56.064906 (+ 121us) client_negotiation.cc:240] Negotiation successful
0926 04:00:56.065091 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":74,"threads_started":1}
I20260926 04:00:56.066051 15125 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46215
I20260926 04:00:56.066159 15125 heartbeater.cc:464] Registering TS with master...
I20260926 04:00:56.066327 15125 heartbeater.cc:511] Master 127.0.98.62:46215 requested a full tablet report, sending...
I20260926 04:00:56.066713 14666 ts_manager.cc:194] Registered new tserver with Master: febbc732bc55422e9ee6bdf4c48eec3a (127.0.98.3:37355)
I20260926 04:00:56.067087 14666 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40949
I20260926 04:00:56.076959 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 04:00:56.079821 15129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:56.077553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45926 (local address 127.0.98.62:46215)
0926 04:00:56.077613 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 04:00:56.077615 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:00:56.077727 (+ 112us) server_negotiation.cc:407] Connection header received
0926 04:00:56.077789 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:00:56.077790 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:00:56.077822 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:00:56.077883 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 04:00:56.077997 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:56.078464 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:56.079184 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:56.079373 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:56.079423 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:56.079554 (+ 131us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 04:00:56.079647 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 04:00:56.079707 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 04:00:56.079737 (+ 30us) server_negotiation.cc:299] Negotiation successful
0926 04:00:56.079759 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 04:00:56.080075 14666 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:45926:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 04:00:56.081722 14666 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 04:00:56.083814 14788 tablet_service.cc:1467] Processing CreateTablet for tablet 1378e8a97f164ac78fea36a2a6d438e1 (DEFAULT_TABLE table=test-table [id=b0acc8b542c145fd9710e57eb26c7b91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:00:56.084021 14788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1378e8a97f164ac78fea36a2a6d438e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:00:56.085459 14746 tablet_bootstrap.cc:492] T 1378e8a97f164ac78fea36a2a6d438e1 P c199a3d5f00d4157bb393acf090b3fda: Bootstrap starting.
I20260926 04:00:56.085845 14746 tablet_bootstrap.cc:654] T 1378e8a97f164ac78fea36a2a6d438e1 P c199a3d5f00d4157bb393acf090b3fda: Neither blocks nor log segments found. Creating new log.
I20260926 04:00:56.086437 14746 tablet_bootstrap.cc:492] T 1378e8a97f164ac78fea36a2a6d438e1 P c199a3d5f00d4157bb393acf090b3fda: No bootstrap required, opened a new log
I20260926 04:00:56.086484 14746 ts_tablet_manager.cc:1397] T 1378e8a97f164ac78fea36a2a6d438e1 P c199a3d5f00d4157bb393acf090b3fda: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 04:00:56.086582 14746 raft_consensus.cc:348] T 1378e8a97f164ac78fea36a2a6d438e1 P c199a3d5f00d4157bb393acf090b3fda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9e9f973c5bf41f0bc6967ce64cc9706" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35223 } } peers { permanent_uuid: "febbc732bc55422e9ee6bdf4c48eec3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37355 } } peers { permanent_uuid: "c199a3d5f00d4157bb393acf090b3fda" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37653 } }
I20260926 04:00:56.086655 14746 raft_consensus.cc:374] T 1378e8a97f164ac78fea36a2a6d438e1 P c199a3d5f00d4157bb393acf090b3fda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:00:56.086702 14746 raft_consensus.cc:729] T 1378e8a97f164ac78fea36a2a6d438e1 P c199a3d5f00d4157bb393acf090b3fda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c199a3d5f00d4157bb393acf090b3fda, State: Initialized, Role: FOLLOWER
I20260926 04:00:56.086742 14746 consensus_queue.cc:260] T 1378e8a97f164ac78fea36a2a6d438e1 P c199a3d5f00d4157bb393acf090b3fda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9e9f973c5bf41f0bc6967ce64cc9706" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35223 } } peers { permanent_uuid: "febbc732bc55422e9ee6bdf4c48eec3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37355 } } peers { permanent_uuid: "c199a3d5f00d4157bb393acf090b3fda" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37653 } }
I20260926 04:00:56.086822 14746 ts_tablet_manager.cc:1428] T 1378e8a97f164ac78fea36a2a6d438e1 P c199a3d5f00d4157bb393acf090b3fda: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 04:00:56.087518 15131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:56.083464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37355 (local address 127.0.0.1:45848)
0926 04:00:56.083598 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 04:00:56.083609 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 04:00:56.083661 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:00:56.084000 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:00:56.084001 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:00:56.084003 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:00:56.084054 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:00:56.084056 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:00:56.085389 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:00:56.085390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:00:56.086242 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:00:56.086265 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:00:56.087223 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:00:56.087224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:00:56.087309 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:56.087313 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 04:00:56.087374 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 04:00:56.087421 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":90,"threads_started":1}
I20260926 04:00:56.087762 15133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:56.083665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45848 (local address 127.0.98.3:37355)
0926 04:00:56.083819 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 04:00:56.083824 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 04:00:56.083834 (+ 10us) server_negotiation.cc:407] Connection header received
0926 04:00:56.083879 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:00:56.083881 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:00:56.083896 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:00:56.084001 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:00:56.084145 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:56.085285 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:56.086367 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:56.087117 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:56.087220 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:56.087277 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 04:00:56.087630 (+ 353us) server_negotiation.cc:299] Negotiation successful
0926 04:00:56.087681 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":91,"threads_started":1}
I20260926 04:00:56.088375 15059 tablet_service.cc:1467] Processing CreateTablet for tablet 1378e8a97f164ac78fea36a2a6d438e1 (DEFAULT_TABLE table=test-table [id=b0acc8b542c145fd9710e57eb26c7b91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:00:56.088565 14861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:56.083445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35223 (local address 127.0.0.1:54878)
0926 04:00:56.083560 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 04:00:56.083571 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 04:00:56.083609 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:00:56.084970 (+ 1361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:00:56.084971 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:00:56.084972 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:00:56.085008 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:00:56.085009 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:00:56.086316 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:00:56.086317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:00:56.087390 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:00:56.087392 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:00:56.088335 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:00:56.088336 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:00:56.088399 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:56.088403 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 04:00:56.088456 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 04:00:56.088499 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96}
I20260926 04:00:56.088585 15132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:56.083565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54878 (local address 127.0.98.2:35223)
0926 04:00:56.084808 (+ 1243us) server_negotiation.cc:206] Beginning negotiation
0926 04:00:56.084812 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 04:00:56.084822 (+ 10us) server_negotiation.cc:407] Connection header received
0926 04:00:56.084859 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:00:56.084861 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:00:56.084875 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:00:56.084951 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:00:56.085074 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:56.086221 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:56.087456 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:56.088229 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:56.088333 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:56.088386 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 04:00:56.088467 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 04:00:56.088504 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1196,"thread_start_us":98,"threads_started":1}
I20260926 04:00:56.088770 15059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1378e8a97f164ac78fea36a2a6d438e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:00:56.089082 14927 tablet_service.cc:1467] Processing CreateTablet for tablet 1378e8a97f164ac78fea36a2a6d438e1 (DEFAULT_TABLE table=test-table [id=b0acc8b542c145fd9710e57eb26c7b91]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:00:56.089346 14927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1378e8a97f164ac78fea36a2a6d438e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:00:56.090987 14885 tablet_bootstrap.cc:492] T 1378e8a97f164ac78fea36a2a6d438e1 P a9e9f973c5bf41f0bc6967ce64cc9706: Bootstrap starting.
I20260926 04:00:56.091661 15135 tablet_bootstrap.cc:492] T 1378e8a97f164ac78fea36a2a6d438e1 P febbc732bc55422e9ee6bdf4c48eec3a: Bootstrap starting.
I20260926 04:00:56.091715 14885 tablet_bootstrap.cc:654] T 1378e8a97f164ac78fea36a2a6d438e1 P a9e9f973c5bf41f0bc6967ce64cc9706: Neither blocks nor log segments found. Creating new log.
I20260926 04:00:56.092031 14885 log.cc:824] T 1378e8a97f164ac78fea36a2a6d438e1 P a9e9f973c5bf41f0bc6967ce64cc9706: Log is configured to *not* fsync() on all Append() calls
I20260926 04:00:56.092368 15135 tablet_bootstrap.cc:654] T 1378e8a97f164ac78fea36a2a6d438e1 P febbc732bc55422e9ee6bdf4c48eec3a: Neither blocks nor log segments found. Creating new log.
I20260926 04:00:56.092754 14885 tablet_bootstrap.cc:492] T 1378e8a97f164ac78fea36a2a6d438e1 P a9e9f973c5bf41f0bc6967ce64cc9706: No bootstrap required, opened a new log
I20260926 04:00:56.092804 14885 ts_tablet_manager.cc:1397] T 1378e8a97f164ac78fea36a2a6d438e1 P a9e9f973c5bf41f0bc6967ce64cc9706: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 04:00:56.092967 15135 log.cc:824] T 1378e8a97f164ac78fea36a2a6d438e1 P febbc732bc55422e9ee6bdf4c48eec3a: Log is configured to *not* fsync() on all Append() calls
I20260926 04:00:56.093633 15135 tablet_bootstrap.cc:492] T 1378e8a97f164ac78fea36a2a6d438e1 P febbc732bc55422e9ee6bdf4c48eec3a: No bootstrap required, opened a new log
I20260926 04:00:56.093719 15135 ts_tablet_manager.cc:1397] T 1378e8a97f164ac78fea36a2a6d438e1 P febbc732bc55422e9ee6bdf4c48eec3a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 04:00:56.094254 14885 raft_consensus.cc:348] T 1378e8a97f164ac78fea36a2a6d438e1 P a9e9f973c5bf41f0bc6967ce64cc9706 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9e9f973c5bf41f0bc6967ce64cc9706" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35223 } } peers { permanent_uuid: "febbc732bc55422e9ee6bdf4c48eec3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37355 } } peers { permanent_uuid: "c199a3d5f00d4157bb393acf090b3fda" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37653 } }
I20260926 04:00:56.094430 14885 raft_consensus.cc:374] T 1378e8a97f164ac78fea36a2a6d438e1 P a9e9f973c5bf41f0bc6967ce64cc9706 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:00:56.094471 14885 raft_consensus.cc:729] T 1378e8a97f164ac78fea36a2a6d438e1 P a9e9f973c5bf41f0bc6967ce64cc9706 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9e9f973c5bf41f0bc6967ce64cc9706, State: Initialized, Role: FOLLOWER
I20260926 04:00:56.094587 14885 consensus_queue.cc:260] T 1378e8a97f164ac78fea36a2a6d438e1 P a9e9f973c5bf41f0bc6967ce64cc9706 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9e9f973c5bf41f0bc6967ce64cc9706" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35223 } } peers { permanent_uuid: "febbc732bc55422e9ee6bdf4c48eec3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37355 } } peers { permanent_uuid: "c199a3d5f00d4157bb393acf090b3fda" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37653 } }
I20260926 04:00:56.094849 14885 ts_tablet_manager.cc:1428] T 1378e8a97f164ac78fea36a2a6d438e1 P a9e9f973c5bf41f0bc6967ce64cc9706: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 04:00:56.094877 14993 heartbeater.cc:503] Master 127.0.98.62:46215 was elected leader, sending a full tablet report...
I20260926 04:00:56.095487 15135 raft_consensus.cc:348] T 1378e8a97f164ac78fea36a2a6d438e1 P febbc732bc55422e9ee6bdf4c48eec3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9e9f973c5bf41f0bc6967ce64cc9706" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35223 } } peers { permanent_uuid: "febbc732bc55422e9ee6bdf4c48eec3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37355 } } peers { permanent_uuid: "c199a3d5f00d4157bb393acf090b3fda" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37653 } }
I20260926 04:00:56.095674 15135 raft_consensus.cc:374] T 1378e8a97f164ac78fea36a2a6d438e1 P febbc732bc55422e9ee6bdf4c48eec3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:00:56.095721 15135 raft_consensus.cc:729] T 1378e8a97f164ac78fea36a2a6d438e1 P febbc732bc55422e9ee6bdf4c48eec3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: febbc732bc55422e9ee6bdf4c48eec3a, State: Initialized, Role: FOLLOWER
I20260926 04:00:56.095839 15135 consensus_queue.cc:260] T 1378e8a97f164ac78fea36a2a6d438e1 P febbc732bc55422e9ee6bdf4c48eec3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9e9f973c5bf41f0bc6967ce64cc9706" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35223 } } peers { permanent_uuid: "febbc732bc55422e9ee6bdf4c48eec3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37355 } } peers { permanent_uuid: "c199a3d5f00d4157bb393acf090b3fda" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37653 } }
I20260926 04:00:56.096122 15135 ts_tablet_manager.cc:1428] T 1378e8a97f164ac78fea36a2a6d438e1 P febbc732bc55422e9ee6bdf4c48eec3a: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260926 04:00:56.096151 15125 heartbeater.cc:503] Master 127.0.98.62:46215 was elected leader, sending a full tablet report...
W20260926 04:00:56.145659 14994 tablet.cc:2369] T 1378e8a97f164ac78fea36a2a6d438e1 P a9e9f973c5bf41f0bc6967ce64cc9706: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 04:00:56.244776 15139 raft_consensus.cc:482] T 1378e8a97f164ac78fea36a2a6d438e1 P febbc732bc55422e9ee6bdf4c48eec3a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 04:00:56.244915 15139 raft_consensus.cc:504] T 1378e8a97f164ac78fea36a2a6d438e1 P febbc732bc55422e9ee6bdf4c48eec3a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9e9f973c5bf41f0bc6967ce64cc9706" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35223 } } peers { permanent_uuid: "febbc732bc55422e9ee6bdf4c48eec3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37355 } } peers { permanent_uuid: "c199a3d5f00d4157bb393acf090b3fda" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37653 } }
I20260926 04:00:56.245281 15139 leader_election.cc:288] T 1378e8a97f164ac78fea36a2a6d438e1 P febbc732bc55422e9ee6bdf4c48eec3a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a9e9f973c5bf41f0bc6967ce64cc9706 (127.0.98.2:35223), c199a3d5f00d4157bb393acf090b3fda (127.0.98.1:37653)
I20260926 04:00:56.248260 15140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:56.245500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35223 (local address 127.0.98.3:49279)
0926 04:00:56.245675 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 04:00:56.245686 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 04:00:56.245754 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:00:56.245893 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:00:56.245894 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:00:56.245895 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:00:56.245992 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:00:56.245994 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:00:56.246620 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:00:56.246621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:00:56.247407 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:00:56.247409 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:00:56.248003 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:00:56.248004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:00:56.248080 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:56.248084 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 04:00:56.248127 (+ 43us) client_negotiation.cc:240] Negotiation successful
0926 04:00:56.248156 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":28,"thread_start_us":97,"threads_started":1}
I20260926 04:00:56.248272 15132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:56.245598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49279 (local address 127.0.98.2:35223)
0926 04:00:56.245641 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 04:00:56.245644 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:00:56.245771 (+ 127us) server_negotiation.cc:407] Connection header received
0926 04:00:56.245801 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:00:56.245802 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:00:56.245813 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:00:56.245878 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:00:56.246061 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:56.246507 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:56.247489 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:56.247917 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:56.247972 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:56.248002 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0926 04:00:56.248152 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 04:00:56.248200 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 04:00:56.248332 14862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:56.245543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42009 (local address 127.0.98.1:37653)
0926 04:00:56.245592 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 04:00:56.245594 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:00:56.245605 (+ 11us) server_negotiation.cc:407] Connection header received
0926 04:00:56.245690 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:00:56.245691 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:00:56.245706 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:00:56.245765 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:00:56.245959 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:56.246487 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:56.247583 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:56.247926 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:56.248014 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:56.248024 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 04:00:56.248214 (+ 190us) server_negotiation.cc:299] Negotiation successful
0926 04:00:56.248253 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 04:00:56.248345 15127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:56.245491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37653 (local address 127.0.98.3:42009)
0926 04:00:56.245554 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0926 04:00:56.245568 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 04:00:56.245615 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:00:56.245791 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:00:56.245792 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:00:56.245794 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:00:56.245871 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:00:56.245873 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:00:56.246578 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:00:56.246580 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:00:56.247493 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:00:56.247495 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:00:56.248036 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:00:56.248037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:00:56.248130 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:56.248133 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 04:00:56.248191 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 04:00:56.248230 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":34}
I20260926 04:00:56.248674 14947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1378e8a97f164ac78fea36a2a6d438e1" candidate_uuid: "febbc732bc55422e9ee6bdf4c48eec3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9e9f973c5bf41f0bc6967ce64cc9706" is_pre_election: true
I20260926 04:00:56.248682 14808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1378e8a97f164ac78fea36a2a6d438e1" candidate_uuid: "febbc732bc55422e9ee6bdf4c48eec3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c199a3d5f00d4157bb393acf090b3fda" is_pre_election: true
I20260926 04:00:56.248828 14947 raft_consensus.cc:2454] T 1378e8a97f164ac78fea36a2a6d438e1 P a9e9f973c5bf41f0bc6967ce64cc9706 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate febbc732bc55422e9ee6bdf4c48eec3a in term 0.
I20260926 04:00:56.248873 14808 raft_consensus.cc:2454] T 1378e8a97f164ac78fea36a2a6d438e1 P c199a3d5f00d4157bb393acf090b3fda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate febbc732bc55422e9ee6bdf4c48eec3a in term 0.
I20260926 04:00:56.249096 15012 leader_election.cc:302] T 1378e8a97f164ac78fea36a2a6d438e1 P febbc732bc55422e9ee6bdf4c48eec3a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9e9f973c5bf41f0bc6967ce64cc9706, febbc732bc55422e9ee6bdf4c48eec3a; no voters:
I20260926 04:00:56.249260 15139 raft_consensus.cc:2781] T 1378e8a97f164ac78fea36a2a6d438e1 P febbc732bc55422e9ee6bdf4c48eec3a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 04:00:56.249333 15139 raft_consensus.cc:482] T 1378e8a97f164ac78fea36a2a6d438e1 P febbc732bc55422e9ee6bdf4c48eec3a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 04:00:56.249354 15139 raft_consensus.cc:3037] T 1378e8a97f164ac78fea36a2a6d438e1 P febbc732bc55422e9ee6bdf4c48eec3a [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:00:56.250097 15139 raft_consensus.cc:504] T 1378e8a97f164ac78fea36a2a6d438e1 P febbc732bc55422e9ee6bdf4c48eec3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9e9f973c5bf41f0bc6967ce64cc9706" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35223 } } peers { permanent_uuid: "febbc732bc55422e9ee6bdf4c48eec3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37355 } } peers { permanent_uuid: "c199a3d5f00d4157bb393acf090b3fda" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37653 } }
I20260926 04:00:56.250238 15139 leader_election.cc:288] T 1378e8a97f164ac78fea36a2a6d438e1 P febbc732bc55422e9ee6bdf4c48eec3a [CANDIDATE]: Term 1 election: Requested vote from peers a9e9f973c5bf41f0bc6967ce64cc9706 (127.0.98.2:35223), c199a3d5f00d4157bb393acf090b3fda (127.0.98.1:37653)
I20260926 04:00:56.250391 14808 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1378e8a97f164ac78fea36a2a6d438e1" candidate_uuid: "febbc732bc55422e9ee6bdf4c48eec3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c199a3d5f00d4157bb393acf090b3fda"
I20260926 04:00:56.250442 14947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1378e8a97f164ac78fea36a2a6d438e1" candidate_uuid: "febbc732bc55422e9ee6bdf4c48eec3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9e9f973c5bf41f0bc6967ce64cc9706"
I20260926 04:00:56.250484 14808 raft_consensus.cc:3037] T 1378e8a97f164ac78fea36a2a6d438e1 P c199a3d5f00d4157bb393acf090b3fda [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:00:56.250528 14947 raft_consensus.cc:3037] T 1378e8a97f164ac78fea36a2a6d438e1 P a9e9f973c5bf41f0bc6967ce64cc9706 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:00:56.251204 14947 raft_consensus.cc:2454] T 1378e8a97f164ac78fea36a2a6d438e1 P a9e9f973c5bf41f0bc6967ce64cc9706 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate febbc732bc55422e9ee6bdf4c48eec3a in term 1.
I20260926 04:00:56.251428 15012 leader_election.cc:302] T 1378e8a97f164ac78fea36a2a6d438e1 P febbc732bc55422e9ee6bdf4c48eec3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9e9f973c5bf41f0bc6967ce64cc9706, febbc732bc55422e9ee6bdf4c48eec3a; no voters:
I20260926 04:00:56.251652 14808 raft_consensus.cc:2454] T 1378e8a97f164ac78fea36a2a6d438e1 P c199a3d5f00d4157bb393acf090b3fda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate febbc732bc55422e9ee6bdf4c48eec3a in term 1.
I20260926 04:00:56.251652 15139 raft_consensus.cc:2781] T 1378e8a97f164ac78fea36a2a6d438e1 P febbc732bc55422e9ee6bdf4c48eec3a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:00:56.251811 15139 raft_consensus.cc:686] T 1378e8a97f164ac78fea36a2a6d438e1 P febbc732bc55422e9ee6bdf4c48eec3a [term 1 LEADER]: Becoming Leader. State: Replica: febbc732bc55422e9ee6bdf4c48eec3a, State: Running, Role: LEADER
I20260926 04:00:56.251914 15139 consensus_queue.cc:237] T 1378e8a97f164ac78fea36a2a6d438e1 P febbc732bc55422e9ee6bdf4c48eec3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9e9f973c5bf41f0bc6967ce64cc9706" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35223 } } peers { permanent_uuid: "febbc732bc55422e9ee6bdf4c48eec3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37355 } } peers { permanent_uuid: "c199a3d5f00d4157bb393acf090b3fda" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37653 } }
W20260926 04:00:56.252908 14855 tablet.cc:2369] T 1378e8a97f164ac78fea36a2a6d438e1 P c199a3d5f00d4157bb393acf090b3fda: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 04:00:56.252846 14666 catalog_manager.cc:5495] T 1378e8a97f164ac78fea36a2a6d438e1 P febbc732bc55422e9ee6bdf4c48eec3a reported cstate change: term changed from 0 to 1, leader changed from <none> to febbc732bc55422e9ee6bdf4c48eec3a (127.0.98.3). New cstate: current_term: 1 leader_uuid: "febbc732bc55422e9ee6bdf4c48eec3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9e9f973c5bf41f0bc6967ce64cc9706" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "febbc732bc55422e9ee6bdf4c48eec3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c199a3d5f00d4157bb393acf090b3fda" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37653 } health_report { overall_health: UNKNOWN } } }
I20260926 04:00:56.272027 15133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:56.269749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45850 (local address 127.0.98.3:37355)
0926 04:00:56.269810 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 04:00:56.269817 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 04:00:56.269825 (+ 8us) server_negotiation.cc:407] Connection header received
0926 04:00:56.269867 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:00:56.269867 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:00:56.269885 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:00:56.269946 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 04:00:56.270062 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:56.270520 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:56.271232 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:56.271410 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:56.271459 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:56.271623 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 04:00:56.271714 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 04:00:56.271772 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 04:00:56.271861 (+ 89us) server_negotiation.cc:299] Negotiation successful
0926 04:00:56.271878 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 04:00:56.273969 14808 raft_consensus.cc:1261] T 1378e8a97f164ac78fea36a2a6d438e1 P c199a3d5f00d4157bb393acf090b3fda [term 1 FOLLOWER]: Refusing update from remote peer febbc732bc55422e9ee6bdf4c48eec3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:00:56.274029 14947 raft_consensus.cc:1261] T 1378e8a97f164ac78fea36a2a6d438e1 P a9e9f973c5bf41f0bc6967ce64cc9706 [term 1 FOLLOWER]: Refusing update from remote peer febbc732bc55422e9ee6bdf4c48eec3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:00:56.274488 15141 consensus_queue.cc:1035] T 1378e8a97f164ac78fea36a2a6d438e1 P febbc732bc55422e9ee6bdf4c48eec3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c199a3d5f00d4157bb393acf090b3fda" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:00:56.274667 15139 consensus_queue.cc:1035] T 1378e8a97f164ac78fea36a2a6d438e1 P febbc732bc55422e9ee6bdf4c48eec3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9e9f973c5bf41f0bc6967ce64cc9706" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:00:56.277021 15146 mvcc.cc:205] Tried to move back new op lower bound from 7333458969695059968 to 7333458969608577024. Current Snapshot: MvccSnapshot[applied={T|T < 7333458969695059968}]
I20260926 04:00:56.277530 15144 mvcc.cc:205] Tried to move back new op lower bound from 7333458969695059968 to 7333458969608577024. Current Snapshot: MvccSnapshot[applied={T|T < 7333458969695059968 or (T in {7333458969695059968})}]
I20260926 04:00:56.279520 14664 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:45926:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 04:00:56.279620 14664 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:45926:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 04:00:56.280411 14664 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a2e82428eba746a8b01d279a4f8b659f: Sending DeleteTablet for 3 replicas of tablet 1378e8a97f164ac78fea36a2a6d438e1
I20260926 04:00:56.280833 15059 tablet_service.cc:1514] Processing DeleteTablet for tablet 1378e8a97f164ac78fea36a2a6d438e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:00:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45848
I20260926 04:00:56.280861 14927 tablet_service.cc:1514] Processing DeleteTablet for tablet 1378e8a97f164ac78fea36a2a6d438e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:00:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54878
I20260926 04:00:56.281059 14788 tablet_service.cc:1514] Processing DeleteTablet for tablet 1378e8a97f164ac78fea36a2a6d438e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:00:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60674
I20260926 04:00:56.281073 15152 tablet_replica.cc:330] T 1378e8a97f164ac78fea36a2a6d438e1 P febbc732bc55422e9ee6bdf4c48eec3a: stopping tablet replica
I20260926 04:00:56.281189 15152 raft_consensus.cc:2229] T 1378e8a97f164ac78fea36a2a6d438e1 P febbc732bc55422e9ee6bdf4c48eec3a [term 1 LEADER]: Raft consensus shutting down.
I20260926 04:00:56.281287 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 14704
I20260926 04:00:56.281309 15152 raft_consensus.cc:2258] T 1378e8a97f164ac78fea36a2a6d438e1 P febbc732bc55422e9ee6bdf4c48eec3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:00:56.281307 15153 tablet_replica.cc:330] T 1378e8a97f164ac78fea36a2a6d438e1 P a9e9f973c5bf41f0bc6967ce64cc9706: stopping tablet replica
I20260926 04:00:56.281399 15153 raft_consensus.cc:2229] T 1378e8a97f164ac78fea36a2a6d438e1 P a9e9f973c5bf41f0bc6967ce64cc9706 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 04:00:56.281492 15153 raft_consensus.cc:2258] T 1378e8a97f164ac78fea36a2a6d438e1 P a9e9f973c5bf41f0bc6967ce64cc9706 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:00:56.282291 15152 ts_tablet_manager.cc:1905] T 1378e8a97f164ac78fea36a2a6d438e1 P febbc732bc55422e9ee6bdf4c48eec3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:00:56.283142 15153 ts_tablet_manager.cc:1905] T 1378e8a97f164ac78fea36a2a6d438e1 P a9e9f973c5bf41f0bc6967ce64cc9706: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:00:56.284693 15152 ts_tablet_manager.cc:1918] T 1378e8a97f164ac78fea36a2a6d438e1 P febbc732bc55422e9ee6bdf4c48eec3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 04:00:56.284766 15152 log.cc:1196] T 1378e8a97f164ac78fea36a2a6d438e1 P febbc732bc55422e9ee6bdf4c48eec3a: Deleting WAL directory at /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-2/wal/wals/1378e8a97f164ac78fea36a2a6d438e1
I20260926 04:00:56.284912 15153 ts_tablet_manager.cc:1918] T 1378e8a97f164ac78fea36a2a6d438e1 P a9e9f973c5bf41f0bc6967ce64cc9706: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 04:00:56.284964 15153 log.cc:1196] T 1378e8a97f164ac78fea36a2a6d438e1 P a9e9f973c5bf41f0bc6967ce64cc9706: Deleting WAL directory at /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-1/wal/wals/1378e8a97f164ac78fea36a2a6d438e1
I20260926 04:00:56.285053 15152 ts_tablet_manager.cc:1939] T 1378e8a97f164ac78fea36a2a6d438e1 P febbc732bc55422e9ee6bdf4c48eec3a: Deleting consensus metadata
I20260926 04:00:56.285211 15153 ts_tablet_manager.cc:1939] T 1378e8a97f164ac78fea36a2a6d438e1 P a9e9f973c5bf41f0bc6967ce64cc9706: Deleting consensus metadata
I20260926 04:00:56.285606 14650 catalog_manager.cc:4841] TS a9e9f973c5bf41f0bc6967ce64cc9706 (127.0.98.2:35223): tablet 1378e8a97f164ac78fea36a2a6d438e1 (table test-table [id=b0acc8b542c145fd9710e57eb26c7b91]) successfully deleted
W20260926 04:00:56.287480 14651 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37653 (error 108)
I20260926 04:00:56.287652 14652 catalog_manager.cc:4841] TS febbc732bc55422e9ee6bdf4c48eec3a (127.0.98.3:37355): tablet 1378e8a97f164ac78fea36a2a6d438e1 (table test-table [id=b0acc8b542c145fd9710e57eb26c7b91]) successfully deleted
I20260926 04:00:56.287878 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 14859
I20260926 04:00:56.289072 14861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:56.288933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37653 (local address 127.0.0.1:60680)
0926 04:00:56.288993 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0926 04:00:56.289032 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 04:00:56.289245 14651 catalog_manager.cc:4584] TS c199a3d5f00d4157bb393acf090b3fda (127.0.98.1:37653): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1378e8a97f164ac78fea36a2a6d438e1: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37653: connect: Connection refused (error 111)
I20260926 04:00:56.294951 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 14996
I20260926 04:00:56.299832 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 14633
I20260926 04:00:56.305238 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46215
--webserver_interface=127.0.98.62
--webserver_port=34903
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46215
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:00:56.385960 15156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:00:56.386219 15156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:00:56.386256 15156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:00:56.387408 15156 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:00:56.387481 15156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:00:56.387506 15156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:00:56.387535 15156 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:00:56.387566 15156 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:00:56.389050 15156 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46215
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=34903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:00:56.389334 15156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:00:56.389556 15156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:00:56.394482 15164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:00:56.394641 15161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:00:56.394747 15156 server_base.cc:1034] running on GCE node
W20260926 04:00:56.394515 15162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:00:56.395051 15156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:00:56.395362 15156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:00:56.396512 15156 hybrid_clock.cc:648] HybridClock initialized: now 1790395256396501 us; error 27 us; skew 500 ppm
Sep 26 04:00:56 dist-test-slave-nhpw krb5kdc[14088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395256, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:00:56.399001 15156 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 04:00:56.399489 15156 webserver.cc:460] Webserver started at http://127.0.98.62:34903/ using document root <none> and password file <none>
I20260926 04:00:56.399739 15156 fs_manager.cc:359] Metadata directory not provided
I20260926 04:00:56.399811 15156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:00:56.401261 15156 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 04:00:56.401906 15171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:00:56.402096 15156 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 04:00:56.402208 15156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/master-0/wal
uuid: "a2e82428eba746a8b01d279a4f8b659f"
format_stamp: "Formatted at 2026-09-26 04:00:54 on dist-test-slave-nhpw"
server_key: "88b3ac9de17d77ea99a6433a2d34f8e7"
server_key_iv: "b1352df44537b6a5c96335f7bf538c1c"
server_key_version: "encryptionkey@0"
I20260926 04:00:56.402572 15156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:00:56.410801 15156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:00:56.412914 15156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:00:56.413061 15156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:00:56.416813 15156 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46215
I20260926 04:00:56.416849 15223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46215 every 8 connection(s)
I20260926 04:00:56.417244 15156 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
I20260926 04:00:56.418910 15224 sys_catalog.cc:263] Verifying existing consensus state
I20260926 04:00:56.419545 15224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2e82428eba746a8b01d279a4f8b659f: Bootstrap starting.
I20260926 04:00:56.419971 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 15156
I20260926 04:00:56.420225 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:37653
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:46215
--webserver_port=34091
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 04:00:56.428228 15224 log.cc:824] T 00000000000000000000000000000000 P a2e82428eba746a8b01d279a4f8b659f: Log is configured to *not* fsync() on all Append() calls
I20260926 04:00:56.430591 15224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2e82428eba746a8b01d279a4f8b659f: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 04:00:56.430897 15224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2e82428eba746a8b01d279a4f8b659f: Bootstrap complete.
I20260926 04:00:56.433174 15224 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2e82428eba746a8b01d279a4f8b659f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2e82428eba746a8b01d279a4f8b659f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46215 } }
I20260926 04:00:56.433425 15224 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2e82428eba746a8b01d279a4f8b659f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2e82428eba746a8b01d279a4f8b659f, State: Initialized, Role: FOLLOWER
I20260926 04:00:56.433629 15224 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2e82428eba746a8b01d279a4f8b659f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2e82428eba746a8b01d279a4f8b659f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46215 } }
I20260926 04:00:56.433727 15224 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2e82428eba746a8b01d279a4f8b659f [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:00:56.433780 15224 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2e82428eba746a8b01d279a4f8b659f [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:00:56.433830 15224 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2e82428eba746a8b01d279a4f8b659f [term 2 FOLLOWER]: Advancing to term 3
I20260926 04:00:56.434793 15224 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2e82428eba746a8b01d279a4f8b659f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2e82428eba746a8b01d279a4f8b659f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46215 } }
I20260926 04:00:56.434909 15224 leader_election.cc:302] T 00000000000000000000000000000000 P a2e82428eba746a8b01d279a4f8b659f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2e82428eba746a8b01d279a4f8b659f; no voters:
I20260926 04:00:56.435105 15224 leader_election.cc:288] T 00000000000000000000000000000000 P a2e82428eba746a8b01d279a4f8b659f [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 04:00:56.435142 15228 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2e82428eba746a8b01d279a4f8b659f [term 3 FOLLOWER]: Leader election won for term 3
I20260926 04:00:56.435259 15228 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2e82428eba746a8b01d279a4f8b659f [term 3 LEADER]: Becoming Leader. State: Replica: a2e82428eba746a8b01d279a4f8b659f, State: Running, Role: LEADER
I20260926 04:00:56.435343 15228 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2e82428eba746a8b01d279a4f8b659f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2e82428eba746a8b01d279a4f8b659f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46215 } }
I20260926 04:00:56.435387 15224 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2e82428eba746a8b01d279a4f8b659f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:00:56.435582 15229 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2e82428eba746a8b01d279a4f8b659f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "a2e82428eba746a8b01d279a4f8b659f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2e82428eba746a8b01d279a4f8b659f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46215 } } }
I20260926 04:00:56.435668 15229 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2e82428eba746a8b01d279a4f8b659f [sys.catalog]: This master's current role is: LEADER
I20260926 04:00:56.435911 15230 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2e82428eba746a8b01d279a4f8b659f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2e82428eba746a8b01d279a4f8b659f. Latest consensus state: current_term: 3 leader_uuid: "a2e82428eba746a8b01d279a4f8b659f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2e82428eba746a8b01d279a4f8b659f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46215 } } }
I20260926 04:00:56.436007 15230 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2e82428eba746a8b01d279a4f8b659f [sys.catalog]: This master's current role is: LEADER
I20260926 04:00:56.436139 15232 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:00:56.436899 15232 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:00:56.436997 15232 catalog_manager.cc:1259] Loaded cluster ID: 8d7e2452f04944cb98926530d868de23
I20260926 04:00:56.437062 15232 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260926 04:00:56.437517 15232 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f9c721b89ad google::LogMessage::Fail() at ??:0
@ 0x7f9c721bb88a google::LogMessage::SendToLog() at ??:0
@ 0x7f9c721b84de google::LogMessage::Flush() at ??:0
@ 0x7f9c721b9a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f9c7a0dd0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f9c7a0e68b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f9c740860d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f9c7407b954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f9c74edf6db start_thread at ??:0
@ 0x7f9c70cb661f clone at ??:0
W20260926 04:00:56.514797 15226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:00:56.515071 15226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:00:56.515111 15226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:00:56.516247 15226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:00:56.516314 15226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:00:56.516360 15226 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 04:00:56.517897 15226 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:37653
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=34091
--tserver_master_addrs=127.0.98.62:46215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15226
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:00:56.518169 15226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:00:56.518409 15226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:00:56.523360 15263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:00:56.523401 15260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:00:56.523602 15226 server_base.cc:1034] running on GCE node
W20260926 04:00:56.523693 15261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:00:56.523917 15226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:00:56.524220 15226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:00:56.525379 15226 hybrid_clock.cc:648] HybridClock initialized: now 1790395256525366 us; error 33 us; skew 500 ppm
Sep 26 04:00:56 dist-test-slave-nhpw krb5kdc[14088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395256, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:00:56.527976 15226 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 04:00:56.528498 15226 webserver.cc:460] Webserver started at http://127.0.98.1:34091/ using document root <none> and password file <none>
I20260926 04:00:56.528805 15226 fs_manager.cc:359] Metadata directory not provided
I20260926 04:00:56.528875 15226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:00:56.530230 15226 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:00:56.530930 15270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:00:56.531114 15226 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:00:56.531180 15226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-0/wal
uuid: "c199a3d5f00d4157bb393acf090b3fda"
format_stamp: "Formatted at 2026-09-26 04:00:54 on dist-test-slave-nhpw"
server_key: "13f7f460d5365dcbd0448ac676ca5d87"
server_key_iv: "75b9a57771d10f436ee080c912d88320"
server_key_version: "encryptionkey@0"
I20260926 04:00:56.531549 15226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:00:56.545506 15226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:00:56.547852 15226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:00:56.547988 15226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:00:56.548244 15226 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:00:56.548708 15277 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 04:00:56.550575 15226 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 04:00:56.550638 15226 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 04:00:56.550686 15226 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 04:00:56.551355 15226 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 04:00:56.551414 15226 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 04:00:56.551492 15277 tablet_bootstrap.cc:492] T 1378e8a97f164ac78fea36a2a6d438e1 P c199a3d5f00d4157bb393acf090b3fda: Bootstrap starting.
I20260926 04:00:56.558676 15226 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37653
I20260926 04:00:56.558763 15384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37653 every 8 connection(s)
I20260926 04:00:56.559113 15226 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
I20260926 04:00:56.562201 15277 log.cc:824] T 1378e8a97f164ac78fea36a2a6d438e1 P c199a3d5f00d4157bb393acf090b3fda: Log is configured to *not* fsync() on all Append() calls
I20260926 04:00:56.564970 15277 tablet_bootstrap.cc:492] T 1378e8a97f164ac78fea36a2a6d438e1 P c199a3d5f00d4157bb393acf090b3fda: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 04:00:56.565285 15277 tablet_bootstrap.cc:492] T 1378e8a97f164ac78fea36a2a6d438e1 P c199a3d5f00d4157bb393acf090b3fda: Bootstrap complete.
I20260926 04:00:56.565696 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 15226
I20260926 04:00:56.565943 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:35223
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:46215
--webserver_port=40201
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 04:00:56.566555 15277 ts_tablet_manager.cc:1397] T 1378e8a97f164ac78fea36a2a6d438e1 P c199a3d5f00d4157bb393acf090b3fda: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.009s
I20260926 04:00:56.567992 15277 raft_consensus.cc:348] T 1378e8a97f164ac78fea36a2a6d438e1 P c199a3d5f00d4157bb393acf090b3fda [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9e9f973c5bf41f0bc6967ce64cc9706" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35223 } } peers { permanent_uuid: "febbc732bc55422e9ee6bdf4c48eec3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37355 } } peers { permanent_uuid: "c199a3d5f00d4157bb393acf090b3fda" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37653 } }
I20260926 04:00:56.568311 15277 raft_consensus.cc:729] T 1378e8a97f164ac78fea36a2a6d438e1 P c199a3d5f00d4157bb393acf090b3fda [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c199a3d5f00d4157bb393acf090b3fda, State: Initialized, Role: FOLLOWER
I20260926 04:00:56.568475 15277 consensus_queue.cc:260] T 1378e8a97f164ac78fea36a2a6d438e1 P c199a3d5f00d4157bb393acf090b3fda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9e9f973c5bf41f0bc6967ce64cc9706" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35223 } } peers { permanent_uuid: "febbc732bc55422e9ee6bdf4c48eec3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37355 } } peers { permanent_uuid: "c199a3d5f00d4157bb393acf090b3fda" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37653 } }
I20260926 04:00:56.568758 15277 ts_tablet_manager.cc:1428] T 1378e8a97f164ac78fea36a2a6d438e1 P c199a3d5f00d4157bb393acf090b3fda: Time spent starting tablet: real 0.002s user 0.003s sys 0.001s
W20260926 04:00:56.651000 15387 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:00:56.559787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.98.1:35381)
0926 04:00:56.559966 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 04:00:56.560009 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 04:00:56.560475 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:00:56.650769 (+ 90294us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":102,"threads_started":1}
W20260926 04:00:56.651255 15385 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46215 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:46215: Client connection negotiation failed: client connection to 127.0.98.62:46215: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 04:00:56.651621 15387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:56.651507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.98.1:49829)
0926 04:00:56.651563 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 04:00:56.651602 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 04:00:56.651939 15387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:56.651861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.98.1:52939)
0926 04:00:56.651903 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0926 04:00:56.651924 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 04:00:56.652060 15385 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 04:00:56.653523 15389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:00:56.653712 15389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:00:56.653764 15389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:00:56.654950 15389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:00:56.654989 15389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:00:56.655045 15389 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 04:00:56.656502 15389 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:35223
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=40201
--tserver_master_addrs=127.0.98.62:46215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15389
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:00:56.656769 15389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:00:56.657014 15389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:00:56.662086 15396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:00:56.662132 15395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:00:56.662070 15389 server_base.cc:1034] running on GCE node
W20260926 04:00:56.662097 15398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:00:56.662519 15389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:00:56.662781 15389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:00:56.663940 15389 hybrid_clock.cc:648] HybridClock initialized: now 1790395256663936 us; error 36 us; skew 500 ppm
Sep 26 04:00:56 dist-test-slave-nhpw krb5kdc[14088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395256, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:00:56.666414 15389 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 04:00:56.666790 15389 webserver.cc:460] Webserver started at http://127.0.98.2:40201/ using document root <none> and password file <none>
I20260926 04:00:56.666983 15389 fs_manager.cc:359] Metadata directory not provided
I20260926 04:00:56.667024 15389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:00:56.668236 15389 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:00:56.668881 15405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:00:56.669054 15389 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 04:00:56.669140 15389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-1/wal
uuid: "a9e9f973c5bf41f0bc6967ce64cc9706"
format_stamp: "Formatted at 2026-09-26 04:00:55 on dist-test-slave-nhpw"
server_key: "0bdff880fe199481e5f395441305c8ea"
server_key_iv: "5ec0f94efb527d4d81c2fc88a306ae2c"
server_key_version: "encryptionkey@0"
I20260926 04:00:56.669500 15389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:00:56.694515 15389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:00:56.696517 15389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:00:56.696712 15389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:00:56.696964 15389 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:00:56.697311 15389 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:00:56.697371 15389 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:00:56.697414 15389 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:00:56.697443 15389 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:00:56.703171 15389 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35223
I20260926 04:00:56.703279 15518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35223 every 8 connection(s)
I20260926 04:00:56.703622 15389 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
I20260926 04:00:56.704788 15521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:56.704321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.98.2:60353)
0926 04:00:56.704499 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 04:00:56.704568 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":97,"threads_started":1}
W20260926 04:00:56.705173 15519 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46215 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:46215: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
I20260926 04:00:56.705476 15521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:56.705354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.98.2:44657)
0926 04:00:56.705426 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0926 04:00:56.705451 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260926 04:00:56.705783 15521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:56.705692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.98.2:40923)
0926 04:00:56.705742 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 04:00:56.705769 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 04:00:56.705924 15519 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 04:00:56.711863 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 15389
I20260926 04:00:56.712154 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:37355
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:46215
--webserver_port=42519
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:00:56.793177 15522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:00:56.793416 15522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:00:56.793453 15522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:00:56.794592 15522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:00:56.794659 15522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:00:56.794704 15522 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 04:00:56.796134 15522 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:37355
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=42519
--tserver_master_addrs=127.0.98.62:46215
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15522
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:00:56.796389 15522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:00:56.796619 15522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:00:56.801445 15528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:00:56.801465 15530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:00:56.801712 15522 server_base.cc:1034] running on GCE node
W20260926 04:00:56.801470 15527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:00:56.801981 15522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:00:56.802217 15522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:00:56.803393 15522 hybrid_clock.cc:648] HybridClock initialized: now 1790395256803375 us; error 30 us; skew 500 ppm
Sep 26 04:00:56 dist-test-slave-nhpw krb5kdc[14088](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395256, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:00:56.805974 15522 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 04:00:56.806401 15522 webserver.cc:460] Webserver started at http://127.0.98.3:42519/ using document root <none> and password file <none>
I20260926 04:00:56.806617 15522 fs_manager.cc:359] Metadata directory not provided
I20260926 04:00:56.806663 15522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:00:56.807932 15522 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:00:56.808521 15537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:00:56.808696 15522 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 04:00:56.808790 15522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-2/wal
uuid: "febbc732bc55422e9ee6bdf4c48eec3a"
format_stamp: "Formatted at 2026-09-26 04:00:55 on dist-test-slave-nhpw"
server_key: "1c1cff0e74cb94c8d1182ad91ac5801f"
server_key_iv: "35d44f9bc936126682151ef75e1c830d"
server_key_version: "encryptionkey@0"
I20260926 04:00:56.809094 15522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:00:56.827666 15522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:00:56.829656 15522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:00:56.829766 15522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:00:56.829952 15522 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:00:56.830259 15522 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:00:56.830300 15522 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:00:56.830324 15522 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:00:56.830370 15522 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:00:56.835992 15522 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37355
I20260926 04:00:56.836054 15650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37355 every 8 connection(s)
I20260926 04:00:56.836400 15522 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
I20260926 04:00:56.837153 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 15522
I20260926 04:00:56.837502 15653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:56.837100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.98.3:45573)
0926 04:00:56.837278 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 04:00:56.837357 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":102,"threads_started":1}
W20260926 04:00:56.837951 15651 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:46215 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:46215: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
I20260926 04:00:56.838212 15653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:56.838124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.98.3:60251)
0926 04:00:56.838168 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 04:00:56.838194 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 04:00:56.838474 15653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:56.838396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.98.3:53907)
0926 04:00:56.838433 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0926 04:00:56.838456 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260926 04:00:56.838593 15651 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 04:00:57.652606 15655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:57.652397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.98.1:52021)
0926 04:00:57.652512 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 04:00:57.652548 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":60,"threads_started":1}
I20260926 04:00:57.706507 15656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:57.706285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.98.2:47325)
0926 04:00:57.706414 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 04:00:57.706455 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":56,"threads_started":1}
I20260926 04:00:57.839250 15657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:57.838876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.98.3:53721)
0926 04:00:57.839011 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 04:00:57.839195 (+ 184us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":55,"threads_started":1}
I20260926 04:00:58.221625 15658 raft_consensus.cc:482] T 1378e8a97f164ac78fea36a2a6d438e1 P c199a3d5f00d4157bb393acf090b3fda [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 04:00:58.221745 15658 raft_consensus.cc:504] T 1378e8a97f164ac78fea36a2a6d438e1 P c199a3d5f00d4157bb393acf090b3fda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9e9f973c5bf41f0bc6967ce64cc9706" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35223 } } peers { permanent_uuid: "febbc732bc55422e9ee6bdf4c48eec3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37355 } } peers { permanent_uuid: "c199a3d5f00d4157bb393acf090b3fda" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37653 } }
I20260926 04:00:58.222060 15658 leader_election.cc:288] T 1378e8a97f164ac78fea36a2a6d438e1 P c199a3d5f00d4157bb393acf090b3fda [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a9e9f973c5bf41f0bc6967ce64cc9706 (127.0.98.2:35223), febbc732bc55422e9ee6bdf4c48eec3a (127.0.98.3:37355)
Sep 26 04:00:58 dist-test-slave-nhpw krb5kdc[14088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395256, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 26 04:00:58 dist-test-slave-nhpw krb5kdc[14088](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395256, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260926 04:00:58.232625 15661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:58.222358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51581 (local address 127.0.98.3:37355)
0926 04:00:58.222537 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0926 04:00:58.222549 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 04:00:58.222612 (+ 63us) server_negotiation.cc:407] Connection header received
0926 04:00:58.223028 (+ 416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:00:58.223030 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:00:58.223066 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:00:58.223249 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:00:58.223955 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:58.224846 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:58.225869 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:58.226253 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:58.226377 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:58.228893 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:00:58.228897 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:00:58.228898 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:00:58.228910 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:00:58.230802 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:00:58.231480 (+ 678us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:00:58.231482 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:00:58.231483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:00:58.231557 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:00:58.231793 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:00:58.231794 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:00:58.231794 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:00:58.232017 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:00:58.232078 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 04:00:58.232390 (+ 312us) server_negotiation.cc:299] Negotiation successful
0926 04:00:58.232508 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":76,"threads_started":1}
I20260926 04:00:58.232641 15662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:58.222390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33167 (local address 127.0.98.2:35223)
0926 04:00:58.222553 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 04:00:58.222564 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 04:00:58.222588 (+ 24us) server_negotiation.cc:407] Connection header received
0926 04:00:58.223012 (+ 424us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:00:58.223014 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:00:58.223053 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:00:58.223186 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:00:58.223952 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:58.224862 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:58.225869 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:00:58.226248 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:00:58.226351 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:58.229418 (+ 3067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:00:58.229421 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:00:58.229421 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:00:58.229430 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:00:58.231149 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:00:58.231480 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:00:58.231482 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:00:58.231482 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:00:58.231556 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:00:58.231820 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:00:58.231821 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:00:58.231821 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:00:58.232029 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:00:58.232129 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 04:00:58.232392 (+ 263us) server_negotiation.cc:299] Negotiation successful
0926 04:00:58.232526 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":58,"threads_started":1}
I20260926 04:00:58.232693 15659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:58.222250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35223 (local address 127.0.98.1:33167)
0926 04:00:58.222381 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 04:00:58.222393 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 04:00:58.222455 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:00:58.223235 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:00:58.223238 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:00:58.223636 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:00:58.223822 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:00:58.223828 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:00:58.224969 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:00:58.224970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:00:58.225763 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:00:58.225765 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:00:58.226333 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:00:58.226334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:00:58.226444 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:58.227036 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:00:58.227047 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:00:58.229078 (+ 2031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:00:58.231249 (+ 2171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:00:58.231249 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:00:58.231250 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:00:58.231401 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:00:58.231651 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:00:58.231652 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:00:58.231652 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:00:58.231735 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:00:58.232130 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:00:58.232131 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:00:58.232223 (+ 92us) client_negotiation.cc:769] Sending connection context
0926 04:00:58.232327 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 04:00:58.232499 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":54,"thread_start_us":80,"threads_started":1}
I20260926 04:00:58.232821 15660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:58.222254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37355 (local address 127.0.98.1:51581)
0926 04:00:58.222419 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 04:00:58.222431 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 04:00:58.222488 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:00:58.223235 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:00:58.223238 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:00:58.223634 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:00:58.223823 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:00:58.223828 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:00:58.224939 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:00:58.224940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:00:58.225762 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:00:58.225764 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:00:58.226364 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:00:58.226365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:00:58.226444 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:00:58.227085 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:00:58.227089 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:00:58.228744 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:00:58.230882 (+ 2138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:00:58.230888 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:00:58.230890 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:00:58.231163 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:00:58.231622 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:00:58.231623 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:00:58.231623 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:00:58.231727 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:00:58.232087 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:00:58.232094 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:00:58.232224 (+ 130us) client_negotiation.cc:769] Sending connection context
0926 04:00:58.232327 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 04:00:58.232579 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":165,"thread_start_us":69,"threads_started":1}
W20260926 04:00:58.233467 15271 leader_election.cc:341] T 1378e8a97f164ac78fea36a2a6d438e1 P c199a3d5f00d4157bb393acf090b3fda [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a9e9f973c5bf41f0bc6967ce64cc9706 (127.0.98.2:35223): Not found: Tablet not found: 1378e8a97f164ac78fea36a2a6d438e1
W20260926 04:00:58.233927 15273 leader_election.cc:341] T 1378e8a97f164ac78fea36a2a6d438e1 P c199a3d5f00d4157bb393acf090b3fda [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer febbc732bc55422e9ee6bdf4c48eec3a (127.0.98.3:37355): Not found: Tablet not found: 1378e8a97f164ac78fea36a2a6d438e1
I20260926 04:00:58.233968 15273 leader_election.cc:302] T 1378e8a97f164ac78fea36a2a6d438e1 P c199a3d5f00d4157bb393acf090b3fda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c199a3d5f00d4157bb393acf090b3fda; no voters: a9e9f973c5bf41f0bc6967ce64cc9706, febbc732bc55422e9ee6bdf4c48eec3a
I20260926 04:00:58.234094 15658 raft_consensus.cc:2726] T 1378e8a97f164ac78fea36a2a6d438e1 P c199a3d5f00d4157bb393acf090b3fda [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 04:00:58.653321 15660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:58.653148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.98.1:58215)
0926 04:00:58.653232 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0926 04:00:58.653272 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 04:00:58.707165 15663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:58.706959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.98.2:43357)
0926 04:00:58.707079 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 04:00:58.707123 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":62,"threads_started":1}
I20260926 04:00:58.839898 15664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:58.839702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.98.3:60081)
0926 04:00:58.839826 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 04:00:58.839861 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":56,"threads_started":1}
I20260926 04:00:59.654069 15665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:59.653833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.98.1:35253)
0926 04:00:59.653975 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 04:00:59.654013 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":62,"threads_started":1}
I20260926 04:00:59.708019 15666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:59.707788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.98.2:34245)
0926 04:00:59.707935 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 04:00:59.707976 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":65,"threads_started":1}
I20260926 04:00:59.840817 15667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:00:59.840519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.98.3:40509)
0926 04:00:59.840712 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 04:00:59.840746 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":80,"threads_started":1}
I20260926 04:01:00.195184 15668 raft_consensus.cc:482] T 1378e8a97f164ac78fea36a2a6d438e1 P c199a3d5f00d4157bb393acf090b3fda [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 04:01:00.195283 15668 raft_consensus.cc:504] T 1378e8a97f164ac78fea36a2a6d438e1 P c199a3d5f00d4157bb393acf090b3fda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9e9f973c5bf41f0bc6967ce64cc9706" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35223 } } peers { permanent_uuid: "febbc732bc55422e9ee6bdf4c48eec3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37355 } } peers { permanent_uuid: "c199a3d5f00d4157bb393acf090b3fda" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37653 } }
I20260926 04:01:00.195453 15668 leader_election.cc:288] T 1378e8a97f164ac78fea36a2a6d438e1 P c199a3d5f00d4157bb393acf090b3fda [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a9e9f973c5bf41f0bc6967ce64cc9706 (127.0.98.2:35223), febbc732bc55422e9ee6bdf4c48eec3a (127.0.98.3:37355)
W20260926 04:01:00.195858 15273 leader_election.cc:341] T 1378e8a97f164ac78fea36a2a6d438e1 P c199a3d5f00d4157bb393acf090b3fda [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer febbc732bc55422e9ee6bdf4c48eec3a (127.0.98.3:37355): Not found: Tablet not found: 1378e8a97f164ac78fea36a2a6d438e1
W20260926 04:01:00.195960 15271 leader_election.cc:341] T 1378e8a97f164ac78fea36a2a6d438e1 P c199a3d5f00d4157bb393acf090b3fda [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a9e9f973c5bf41f0bc6967ce64cc9706 (127.0.98.2:35223): Not found: Tablet not found: 1378e8a97f164ac78fea36a2a6d438e1
I20260926 04:01:00.196035 15271 leader_election.cc:302] T 1378e8a97f164ac78fea36a2a6d438e1 P c199a3d5f00d4157bb393acf090b3fda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c199a3d5f00d4157bb393acf090b3fda; no voters: a9e9f973c5bf41f0bc6967ce64cc9706, febbc732bc55422e9ee6bdf4c48eec3a
I20260926 04:01:00.196132 15668 raft_consensus.cc:2726] T 1378e8a97f164ac78fea36a2a6d438e1 P c199a3d5f00d4157bb393acf090b3fda [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 04:01:00.654812 15669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:00.654595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.98.1:35949)
0926 04:01:00.654717 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 04:01:00.654751 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":65,"threads_started":1}
I20260926 04:01:00.708869 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:00.708580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.98.2:45803)
0926 04:01:00.708757 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 04:01:00.708796 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":122,"threads_started":1}
I20260926 04:01:00.841609 15671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:00.841337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.98.3:36395)
0926 04:01:00.841522 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 04:01:00.841564 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":107,"threads_started":1}
I20260926 04:01:01.655551 15672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:01.655336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.98.1:39485)
0926 04:01:01.655470 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 04:01:01.655507 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":64,"threads_started":1}
I20260926 04:01:01.709582 15673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:01.709366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.98.2:52151)
0926 04:01:01.709498 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 04:01:01.709540 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":60,"threads_started":1}
I20260926 04:01:01.842312 15674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:01.842112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.98.3:39351)
0926 04:01:01.842241 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 04:01:01.842270 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":60,"threads_started":1}
I20260926 04:01:02.656234 15675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:02.656005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.98.1:53343)
0926 04:01:02.656153 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 04:01:02.656188 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":76,"threads_started":1}
I20260926 04:01:02.710278 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:02.710064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.98.2:45657)
0926 04:01:02.710199 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 04:01:02.710234 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":76,"threads_started":1}
I20260926 04:01:02.843149 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:02.842856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.98.3:42381)
0926 04:01:02.843061 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0926 04:01:02.843106 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":114,"threads_started":1}
I20260926 04:01:03.129679 15678 raft_consensus.cc:482] T 1378e8a97f164ac78fea36a2a6d438e1 P c199a3d5f00d4157bb393acf090b3fda [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 04:01:03.129812 15678 raft_consensus.cc:504] T 1378e8a97f164ac78fea36a2a6d438e1 P c199a3d5f00d4157bb393acf090b3fda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9e9f973c5bf41f0bc6967ce64cc9706" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35223 } } peers { permanent_uuid: "febbc732bc55422e9ee6bdf4c48eec3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37355 } } peers { permanent_uuid: "c199a3d5f00d4157bb393acf090b3fda" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37653 } }
I20260926 04:01:03.129981 15678 leader_election.cc:288] T 1378e8a97f164ac78fea36a2a6d438e1 P c199a3d5f00d4157bb393acf090b3fda [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a9e9f973c5bf41f0bc6967ce64cc9706 (127.0.98.2:35223), febbc732bc55422e9ee6bdf4c48eec3a (127.0.98.3:37355)
W20260926 04:01:03.130419 15271 leader_election.cc:341] T 1378e8a97f164ac78fea36a2a6d438e1 P c199a3d5f00d4157bb393acf090b3fda [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a9e9f973c5bf41f0bc6967ce64cc9706 (127.0.98.2:35223): Not found: Tablet not found: 1378e8a97f164ac78fea36a2a6d438e1
W20260926 04:01:03.130504 15273 leader_election.cc:341] T 1378e8a97f164ac78fea36a2a6d438e1 P c199a3d5f00d4157bb393acf090b3fda [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer febbc732bc55422e9ee6bdf4c48eec3a (127.0.98.3:37355): Not found: Tablet not found: 1378e8a97f164ac78fea36a2a6d438e1
I20260926 04:01:03.130566 15273 leader_election.cc:302] T 1378e8a97f164ac78fea36a2a6d438e1 P c199a3d5f00d4157bb393acf090b3fda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c199a3d5f00d4157bb393acf090b3fda; no voters: a9e9f973c5bf41f0bc6967ce64cc9706, febbc732bc55422e9ee6bdf4c48eec3a
I20260926 04:01:03.130661 15678 raft_consensus.cc:2726] T 1378e8a97f164ac78fea36a2a6d438e1 P c199a3d5f00d4157bb393acf090b3fda [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 04:01:03.657002 15679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:03.656785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.98.1:37221)
0926 04:01:03.656921 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 04:01:03.656956 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":76,"threads_started":1}
I20260926 04:01:03.711019 15680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:03.710753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.98.2:40845)
0926 04:01:03.710924 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 04:01:03.710976 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":57,"threads_started":1}
I20260926 04:01:03.843855 15681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:03.843639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.98.3:35959)
0926 04:01:03.843772 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 04:01:03.843810 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":58,"threads_started":1}
I20260926 04:01:04.657781 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:04.657548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.98.1:41857)
0926 04:01:04.657693 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 04:01:04.657735 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":97,"threads_started":1}
I20260926 04:01:04.711674 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:04.711476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.98.2:36555)
0926 04:01:04.711595 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0926 04:01:04.711630 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":54,"threads_started":1}
I20260926 04:01:04.844623 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:04.844381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.98.3:50961)
0926 04:01:04.844536 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 04:01:04.844577 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":80,"threads_started":1}
I20260926 04:01:05.658489 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:05.658283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.98.1:37661)
0926 04:01:05.658407 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 04:01:05.658443 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":71,"threads_started":1}
I20260926 04:01:05.712407 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:05.712202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.98.2:55041)
0926 04:01:05.712324 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 04:01:05.712365 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":59,"threads_started":1}
I20260926 04:01:05.810197 15687 raft_consensus.cc:482] T 1378e8a97f164ac78fea36a2a6d438e1 P c199a3d5f00d4157bb393acf090b3fda [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 04:01:05.810282 15687 raft_consensus.cc:504] T 1378e8a97f164ac78fea36a2a6d438e1 P c199a3d5f00d4157bb393acf090b3fda [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9e9f973c5bf41f0bc6967ce64cc9706" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35223 } } peers { permanent_uuid: "febbc732bc55422e9ee6bdf4c48eec3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37355 } } peers { permanent_uuid: "c199a3d5f00d4157bb393acf090b3fda" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37653 } }
I20260926 04:01:05.810492 15687 leader_election.cc:288] T 1378e8a97f164ac78fea36a2a6d438e1 P c199a3d5f00d4157bb393acf090b3fda [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a9e9f973c5bf41f0bc6967ce64cc9706 (127.0.98.2:35223), febbc732bc55422e9ee6bdf4c48eec3a (127.0.98.3:37355)
W20260926 04:01:05.810866 15273 leader_election.cc:341] T 1378e8a97f164ac78fea36a2a6d438e1 P c199a3d5f00d4157bb393acf090b3fda [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer febbc732bc55422e9ee6bdf4c48eec3a (127.0.98.3:37355): Not found: Tablet not found: 1378e8a97f164ac78fea36a2a6d438e1
W20260926 04:01:05.810933 15271 leader_election.cc:341] T 1378e8a97f164ac78fea36a2a6d438e1 P c199a3d5f00d4157bb393acf090b3fda [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a9e9f973c5bf41f0bc6967ce64cc9706 (127.0.98.2:35223): Not found: Tablet not found: 1378e8a97f164ac78fea36a2a6d438e1
I20260926 04:01:05.811000 15271 leader_election.cc:302] T 1378e8a97f164ac78fea36a2a6d438e1 P c199a3d5f00d4157bb393acf090b3fda [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c199a3d5f00d4157bb393acf090b3fda; no voters: a9e9f973c5bf41f0bc6967ce64cc9706, febbc732bc55422e9ee6bdf4c48eec3a
I20260926 04:01:05.811056 15687 raft_consensus.cc:2726] T 1378e8a97f164ac78fea36a2a6d438e1 P c199a3d5f00d4157bb393acf090b3fda [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 04:01:05.845364 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:05.845158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.98.3:55873)
0926 04:01:05.845284 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 04:01:05.845320 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":61,"threads_started":1}
I20260926 04:01:06.659391 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:06.659019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.98.1:35029)
0926 04:01:06.659312 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0926 04:01:06.659343 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":220,"threads_started":1}
I20260926 04:01:06.713016 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:06.712809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.98.2:49589)
0926 04:01:06.712943 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 04:01:06.712974 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":74,"threads_started":1}
I20260926 04:01:06.846030 15691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:06.845810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.98.3:53917)
0926 04:01:06.845950 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 04:01:06.845987 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":59,"threads_started":1}
I20260926 04:01:07.660111 15692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:07.659898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.98.1:38307)
0926 04:01:07.660029 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 04:01:07.660064 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":63,"threads_started":1}
I20260926 04:01:07.713681 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:07.713467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.98.2:34475)
0926 04:01:07.713602 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 04:01:07.713640 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":75,"threads_started":1}
I20260926 04:01:07.846732 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:07.846510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.98.3:35493)
0926 04:01:07.846653 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 04:01:07.846689 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":84,"threads_started":1}
I20260926 04:01:08.660821 15695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:08.660555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.98.1:56253)
0926 04:01:08.660734 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 04:01:08.660768 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":125,"threads_started":1}
I20260926 04:01:08.714313 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:08.714122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.98.2:50115)
0926 04:01:08.714238 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 04:01:08.714270 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":60,"threads_started":1}
I20260926 04:01:08.847385 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:08.847187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.98.3:52993)
0926 04:01:08.847309 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 04:01:08.847338 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":61,"threads_started":1}
I20260926 04:01:09.661530 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:09.661312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.98.1:39475)
0926 04:01:09.661445 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 04:01:09.661485 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":85,"threads_started":1}
I20260926 04:01:09.714996 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:09.714770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.98.2:54891)
0926 04:01:09.714917 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 04:01:09.714953 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":64,"threads_started":1}
I20260926 04:01:09.848136 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:09.847917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.98.3:38047)
0926 04:01:09.848051 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 04:01:09.848088 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":72,"threads_started":1}
I20260926 04:01:10.662228 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:10.662014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.98.1:39773)
0926 04:01:10.662147 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 04:01:10.662184 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":61,"threads_started":1}
I20260926 04:01:10.715687 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:10.715472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.98.2:39883)
0926 04:01:10.715605 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 04:01:10.715641 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":61,"threads_started":1}
I20260926 04:01:10.848892 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:10.848603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.98.3:44999)
0926 04:01:10.848813 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 04:01:10.848846 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":140,"threads_started":1}
I20260926 04:01:11.663048 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:11.662849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.98.1:49595)
0926 04:01:11.662964 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 04:01:11.663001 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":62,"threads_started":1}
I20260926 04:01:11.716408 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:11.716201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.98.2:33417)
0926 04:01:11.716329 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 04:01:11.716363 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":68,"threads_started":1}
I20260926 04:01:11.842350 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 15226
I20260926 04:01:11.847481 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 15389
I20260926 04:01:11.849696 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:11.849507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46215 (local address 127.0.98.3:37475)
0926 04:01:11.849620 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0926 04:01:11.849658 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:46215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":69,"threads_started":1}
I20260926 04:01:11.851845 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 15522
2026-09-26T04:01:11Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18396 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 04:01:11 dist-test-slave-nhpw krb5kdc[15709](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 04:01:11 dist-test-slave-nhpw krb5kdc[15709](info): set up 2 sockets
Sep 26 04:01:11 dist-test-slave-nhpw krb5kdc[15709](info): commencing operation
krb5kdc: starting...
W20260926 04:01:12.892882 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/krb5kdc/test-user.keytab.
Sep 26 04:01:12 dist-test-slave-nhpw krb5kdc[15709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T04:01:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:01:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 04:01:13.012498 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32815
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32815
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:01:13.092291 15722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:01:13.092499 15722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:01:13.092522 15722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:01:13.093696 15722 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 04:01:13.093739 15722 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 04:01:13.093751 15722 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 04:01:13.093762 15722 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:01:13.093775 15722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:01:13.093788 15722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:01:13.093801 15722 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:01:13.093812 15722 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:01:13.095207 15722 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32815
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:01:13.095391 15722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:01:13.095556 15722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 04:01:13.100723 15722 server_base.cc:1034] running on GCE node
W20260926 04:01:13.100756 15728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:01:13.100754 15730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:01:13.100852 15727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:01:13.101090 15722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:01:13.101374 15722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:01:13.102555 15722 hybrid_clock.cc:648] HybridClock initialized: now 1790395273102546 us; error 30 us; skew 500 ppm
Sep 26 04:01:13 dist-test-slave-nhpw krb5kdc[15709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395273, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:01:13.105096 15722 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 04:01:13.105551 15722 webserver.cc:460] Webserver started at http://127.0.98.62:40033/ using document root <none> and password file <none>
I20260926 04:01:13.105732 15722 fs_manager.cc:359] Metadata directory not provided
I20260926 04:01:13.105772 15722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:01:13.105858 15722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:01:13.106858 15722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/master-0/data/instance:
uuid: "5ad1f0dd809a452ca0355fe421f1b3ad"
format_stamp: "Formatted at 2026-09-26 04:01:13 on dist-test-slave-nhpw"
server_key: "e81730a0b3966cb0eb8ead6f01601210"
server_key_iv: "08b0aaa8da8b10f6beb5f6eca5a53cf2"
server_key_version: "encryptionkey@0"
I20260926 04:01:13.107152 15722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/master-0/wal/instance:
uuid: "5ad1f0dd809a452ca0355fe421f1b3ad"
format_stamp: "Formatted at 2026-09-26 04:01:13 on dist-test-slave-nhpw"
server_key: "e81730a0b3966cb0eb8ead6f01601210"
server_key_iv: "08b0aaa8da8b10f6beb5f6eca5a53cf2"
server_key_version: "encryptionkey@0"
I20260926 04:01:13.108419 15722 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260926 04:01:13.109194 15737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:01:13.109386 15722 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 04:01:13.109459 15722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/master-0/wal
uuid: "5ad1f0dd809a452ca0355fe421f1b3ad"
format_stamp: "Formatted at 2026-09-26 04:01:13 on dist-test-slave-nhpw"
server_key: "e81730a0b3966cb0eb8ead6f01601210"
server_key_iv: "08b0aaa8da8b10f6beb5f6eca5a53cf2"
server_key_version: "encryptionkey@0"
I20260926 04:01:13.109568 15722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:01:13.133700 15722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:01:13.136070 15722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:01:13.136232 15722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:01:13.140491 15722 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32815
I20260926 04:01:13.140545 15789 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32815 every 8 connection(s)
I20260926 04:01:13.140908 15722 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
I20260926 04:01:13.141767 15790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:01:13.144016 15790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ad1f0dd809a452ca0355fe421f1b3ad: Bootstrap starting.
I20260926 04:01:13.144616 15790 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ad1f0dd809a452ca0355fe421f1b3ad: Neither blocks nor log segments found. Creating new log.
I20260926 04:01:13.144987 15790 log.cc:824] T 00000000000000000000000000000000 P 5ad1f0dd809a452ca0355fe421f1b3ad: Log is configured to *not* fsync() on all Append() calls
I20260926 04:01:13.145646 15790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ad1f0dd809a452ca0355fe421f1b3ad: No bootstrap required, opened a new log
I20260926 04:01:13.147161 15790 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ad1f0dd809a452ca0355fe421f1b3ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ad1f0dd809a452ca0355fe421f1b3ad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32815 } }
I20260926 04:01:13.147358 15790 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ad1f0dd809a452ca0355fe421f1b3ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:01:13.147400 15790 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ad1f0dd809a452ca0355fe421f1b3ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ad1f0dd809a452ca0355fe421f1b3ad, State: Initialized, Role: FOLLOWER
I20260926 04:01:13.147457 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 15722
I20260926 04:01:13.147535 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/master-0/wal/instance
I20260926 04:01:13.147598 15790 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ad1f0dd809a452ca0355fe421f1b3ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ad1f0dd809a452ca0355fe421f1b3ad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32815 } }
I20260926 04:01:13.147691 15790 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ad1f0dd809a452ca0355fe421f1b3ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:01:13.147723 15790 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ad1f0dd809a452ca0355fe421f1b3ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:01:13.147739 392 external_mini_cluster.cc:1428] Setting key c23d1a8a99bc469ac1a487452b4a383a
I20260926 04:01:13.147773 15790 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ad1f0dd809a452ca0355fe421f1b3ad [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:01:13.148458 15790 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ad1f0dd809a452ca0355fe421f1b3ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ad1f0dd809a452ca0355fe421f1b3ad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32815 } }
I20260926 04:01:13.148564 15790 leader_election.cc:302] T 00000000000000000000000000000000 P 5ad1f0dd809a452ca0355fe421f1b3ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ad1f0dd809a452ca0355fe421f1b3ad; no voters:
I20260926 04:01:13.148793 15790 leader_election.cc:288] T 00000000000000000000000000000000 P 5ad1f0dd809a452ca0355fe421f1b3ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 04:01:13.148850 15795 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ad1f0dd809a452ca0355fe421f1b3ad [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:01:13.148977 15795 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ad1f0dd809a452ca0355fe421f1b3ad [term 1 LEADER]: Becoming Leader. State: Replica: 5ad1f0dd809a452ca0355fe421f1b3ad, State: Running, Role: LEADER
I20260926 04:01:13.149046 15790 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ad1f0dd809a452ca0355fe421f1b3ad [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:01:13.149152 15795 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ad1f0dd809a452ca0355fe421f1b3ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ad1f0dd809a452ca0355fe421f1b3ad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32815 } }
I20260926 04:01:13.149721 15796 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ad1f0dd809a452ca0355fe421f1b3ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ad1f0dd809a452ca0355fe421f1b3ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ad1f0dd809a452ca0355fe421f1b3ad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32815 } } }
I20260926 04:01:13.149729 15797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ad1f0dd809a452ca0355fe421f1b3ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ad1f0dd809a452ca0355fe421f1b3ad. Latest consensus state: current_term: 1 leader_uuid: "5ad1f0dd809a452ca0355fe421f1b3ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ad1f0dd809a452ca0355fe421f1b3ad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32815 } } }
I20260926 04:01:13.149912 15797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ad1f0dd809a452ca0355fe421f1b3ad [sys.catalog]: This master's current role is: LEADER
I20260926 04:01:13.150159 15796 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ad1f0dd809a452ca0355fe421f1b3ad [sys.catalog]: This master's current role is: LEADER
I20260926 04:01:13.150159 15804 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:01:13.150744 15804 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:01:13.152382 15804 catalog_manager.cc:1347] Generated new cluster ID: f52174d07b3f4bb79225777d91d151b2
I20260926 04:01:13.152432 15804 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 04:01:13 dist-test-slave-nhpw krb5kdc[15709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395272, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 04:01:13.161087 15794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:13.148225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40364 (local address 127.0.98.62:32815)
0926 04:01:13.148708 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0926 04:01:13.148720 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 04:01:13.148750 (+ 30us) server_negotiation.cc:407] Connection header received
0926 04:01:13.148880 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:01:13.148887 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:01:13.148950 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:01:13.149127 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:01:13.150447 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:01:13.152046 (+ 1599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:01:13.153118 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:01:13.153706 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:01:13.153804 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:01:13.156560 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:01:13.156566 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:01:13.156566 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:01:13.156579 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:01:13.159238 (+ 2659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:01:13.159615 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:01:13.159618 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:01:13.159618 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:01:13.159723 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:01:13.159951 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:01:13.159953 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:01:13.159953 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:01:13.160281 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:01:13.160421 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0926 04:01:13.160722 (+ 301us) server_negotiation.cc:299] Negotiation successful
0926 04:01:13.160861 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":436,"thread_start_us":407,"threads_started":1}
I20260926 04:01:13.166940 15804 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:01:13.167448 15804 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:01:13.171047 15804 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ad1f0dd809a452ca0355fe421f1b3ad: Generated new TSK 0
I20260926 04:01:13.171217 15804 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 04:01:13.199631 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32815
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:01:13.280282 15816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:01:13.280498 15816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:01:13.280524 15816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:01:13.281713 15816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:01:13.281759 15816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:01:13.281785 15816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 04:01:13.283254 15816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:01:13.283484 15816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:01:13.283744 15816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:01:13.288724 15822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:01:13.288733 15824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:01:13.288736 15821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:01:13.288897 15816 server_base.cc:1034] running on GCE node
I20260926 04:01:13.289129 15816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:01:13.289364 15816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:01:13.290535 15816 hybrid_clock.cc:648] HybridClock initialized: now 1790395273290519 us; error 37 us; skew 500 ppm
Sep 26 04:01:13 dist-test-slave-nhpw krb5kdc[15709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395273, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:01:13.293139 15816 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 04:01:13.293596 15816 webserver.cc:460] Webserver started at http://127.0.98.1:38119/ using document root <none> and password file <none>
I20260926 04:01:13.293823 15816 fs_manager.cc:359] Metadata directory not provided
I20260926 04:01:13.293869 15816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:01:13.293957 15816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:01:13.294936 15816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-0/data/instance:
uuid: "ffccccc56a0949a29ec497d810c39c43"
format_stamp: "Formatted at 2026-09-26 04:01:13 on dist-test-slave-nhpw"
server_key: "88374dfc5922c3417e129b86cdc294a1"
server_key_iv: "b4eb3d1e4dc1023e48409622dae899da"
server_key_version: "encryptionkey@0"
I20260926 04:01:13.295244 15816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ffccccc56a0949a29ec497d810c39c43"
format_stamp: "Formatted at 2026-09-26 04:01:13 on dist-test-slave-nhpw"
server_key: "88374dfc5922c3417e129b86cdc294a1"
server_key_iv: "b4eb3d1e4dc1023e48409622dae899da"
server_key_version: "encryptionkey@0"
I20260926 04:01:13.296481 15816 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 04:01:13.297303 15831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:01:13.297487 15816 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 04:01:13.297576 15816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-0/wal
uuid: "ffccccc56a0949a29ec497d810c39c43"
format_stamp: "Formatted at 2026-09-26 04:01:13 on dist-test-slave-nhpw"
server_key: "88374dfc5922c3417e129b86cdc294a1"
server_key_iv: "b4eb3d1e4dc1023e48409622dae899da"
server_key_version: "encryptionkey@0"
I20260926 04:01:13.297650 15816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:01:13.323279 15816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:01:13.325736 15816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:01:13.325882 15816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:01:13.326139 15816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:01:13.326447 15816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:01:13.326500 15816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:01:13.326524 15816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:01:13.326536 15816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:01:13.332348 15816 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39669
I20260926 04:01:13.332396 15944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39669 every 8 connection(s)
I20260926 04:01:13.332769 15816 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
I20260926 04:01:13.334367 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 15816
I20260926 04:01:13.334483 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-0/wal/instance
I20260926 04:01:13.334697 392 external_mini_cluster.cc:1428] Setting key a21d67d67308e96b5438b1ace7e8be8b
Sep 26 04:01:13 dist-test-slave-nhpw krb5kdc[15709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395273, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 04:01:13.343673 15794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:13.333471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45073 (local address 127.0.98.62:32815)
0926 04:01:13.333541 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 04:01:13.333543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:01:13.333751 (+ 208us) server_negotiation.cc:407] Connection header received
0926 04:01:13.334325 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:01:13.334326 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:01:13.334340 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:01:13.334404 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:01:13.335573 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:01:13.336089 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:01:13.337486 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:01:13.337826 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:01:13.337903 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:01:13.340561 (+ 2658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:01:13.340562 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:01:13.340563 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:01:13.340570 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:01:13.342176 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:01:13.342638 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:01:13.342639 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:01:13.342640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:01:13.342709 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:01:13.342992 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:01:13.342993 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:01:13.342994 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:01:13.343120 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:01:13.343200 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 04:01:13.343487 (+ 287us) server_negotiation.cc:299] Negotiation successful
0926 04:01:13.343578 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 04:01:13.343883 15947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:13.333456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32815 (local address 127.0.98.1:45073)
0926 04:01:13.333637 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 04:01:13.333673 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 04:01:13.334187 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:01:13.334488 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:01:13.334493 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:01:13.335245 (+ 752us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:01:13.335479 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:01:13.335484 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:01:13.336186 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:01:13.336187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:01:13.337388 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:01:13.337391 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:01:13.337937 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:01:13.337938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:01:13.338116 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:01:13.338682 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:01:13.338694 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:01:13.340452 (+ 1758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:01:13.342293 (+ 1841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:01:13.342301 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:01:13.342303 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:01:13.342551 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:01:13.342790 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:01:13.342790 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:01:13.342791 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:01:13.342914 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:01:13.343189 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:01:13.343195 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:01:13.343339 (+ 144us) client_negotiation.cc:769] Sending connection context
0926 04:01:13.343466 (+ 127us) client_negotiation.cc:240] Negotiation successful
0926 04:01:13.343630 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":87,"threads_started":1}
I20260926 04:01:13.344362 15945 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32815
I20260926 04:01:13.344485 15945 heartbeater.cc:464] Registering TS with master...
I20260926 04:01:13.344717 15945 heartbeater.cc:511] Master 127.0.98.62:32815 requested a full tablet report, sending...
I20260926 04:01:13.345302 15754 ts_manager.cc:194] Registered new tserver with Master: ffccccc56a0949a29ec497d810c39c43 (127.0.98.1:39669)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260926 04:01:13.346024 15754 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:45073
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 04:01:13.361384 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32815
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:01:13.441792 15950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:01:13.442013 15950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:01:13.442036 15950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:01:13.443216 15950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:01:13.443259 15950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:01:13.443284 15950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 04:01:13.444758 15950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15950
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:01:13.445024 15950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:01:13.445256 15950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:01:13.450249 15955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:01:13.450260 15956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:01:13.450260 15958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:01:13.450563 15950 server_base.cc:1034] running on GCE node
I20260926 04:01:13.450798 15950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:01:13.451048 15950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:01:13.452225 15950 hybrid_clock.cc:648] HybridClock initialized: now 1790395273452212 us; error 30 us; skew 500 ppm
Sep 26 04:01:13 dist-test-slave-nhpw krb5kdc[15709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395273, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:01:13.454900 15950 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 04:01:13.455353 15950 webserver.cc:460] Webserver started at http://127.0.98.2:46101/ using document root <none> and password file <none>
I20260926 04:01:13.455587 15950 fs_manager.cc:359] Metadata directory not provided
I20260926 04:01:13.455655 15950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:01:13.455824 15950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:01:13.456951 15950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-1/data/instance:
uuid: "c3b09a5fb725437ca6c07e2fedf6248d"
format_stamp: "Formatted at 2026-09-26 04:01:13 on dist-test-slave-nhpw"
server_key: "c288f9c4922358c61463c80840e6fc18"
server_key_iv: "6460d5e64d64769288126e4a2799ccb7"
server_key_version: "encryptionkey@0"
I20260926 04:01:13.457326 15950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c3b09a5fb725437ca6c07e2fedf6248d"
format_stamp: "Formatted at 2026-09-26 04:01:13 on dist-test-slave-nhpw"
server_key: "c288f9c4922358c61463c80840e6fc18"
server_key_iv: "6460d5e64d64769288126e4a2799ccb7"
server_key_version: "encryptionkey@0"
I20260926 04:01:13.458635 15950 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 04:01:13.459328 15965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:01:13.459498 15950 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 04:01:13.459589 15950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-1/wal
uuid: "c3b09a5fb725437ca6c07e2fedf6248d"
format_stamp: "Formatted at 2026-09-26 04:01:13 on dist-test-slave-nhpw"
server_key: "c288f9c4922358c61463c80840e6fc18"
server_key_iv: "6460d5e64d64769288126e4a2799ccb7"
server_key_version: "encryptionkey@0"
I20260926 04:01:13.459667 15950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:01:13.470072 15950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:01:13.472182 15950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:01:13.472342 15950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:01:13.472573 15950 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:01:13.472920 15950 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:01:13.472977 15950 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:01:13.473018 15950 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:01:13.473044 15950 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:01:13.478756 15950 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41269
I20260926 04:01:13.478832 16078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41269 every 8 connection(s)
I20260926 04:01:13.479146 15950 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
Sep 26 04:01:13 dist-test-slave-nhpw krb5kdc[15709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395273, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 04:01:13.486263 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 15950
I20260926 04:01:13.486377 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-1/wal/instance
I20260926 04:01:13.486908 392 external_mini_cluster.cc:1428] Setting key e8a2d3eeb80972ec3e49e2226accd632
I20260926 04:01:13.488826 15794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:13.479887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43569 (local address 127.0.98.62:32815)
0926 04:01:13.479953 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 04:01:13.479955 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:01:13.480135 (+ 180us) server_negotiation.cc:407] Connection header received
0926 04:01:13.480615 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:01:13.480616 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:01:13.480625 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:01:13.480726 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:01:13.481620 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:01:13.482111 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:01:13.483079 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:01:13.483261 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:01:13.483312 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:01:13.485471 (+ 2159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:01:13.485472 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:01:13.485473 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:01:13.485479 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:01:13.487258 (+ 1779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:01:13.487790 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:01:13.487791 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:01:13.487791 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:01:13.487860 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:01:13.488118 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:01:13.488119 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:01:13.488119 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:01:13.488243 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:01:13.488321 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 04:01:13.488619 (+ 298us) server_negotiation.cc:299] Negotiation successful
0926 04:01:13.488738 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 04:01:13.489101 16081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:13.479872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32815 (local address 127.0.98.2:43569)
0926 04:01:13.480016 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 04:01:13.480056 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 04:01:13.480534 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:01:13.480796 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:01:13.480800 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:01:13.481351 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:01:13.481540 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:01:13.481544 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:01:13.482192 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:01:13.482193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:01:13.483002 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:01:13.483004 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:01:13.483359 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:01:13.483361 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:01:13.483481 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:01:13.483948 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:01:13.483956 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:01:13.485367 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:01:13.487355 (+ 1988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:01:13.487361 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:01:13.487363 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:01:13.487700 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:01:13.487929 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:01:13.487930 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:01:13.487930 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:01:13.488037 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:01:13.488310 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:01:13.488316 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:01:13.488485 (+ 169us) client_negotiation.cc:769] Sending connection context
0926 04:01:13.488597 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 04:01:13.488775 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":75,"threads_started":1}
I20260926 04:01:13.489615 16079 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32815
I20260926 04:01:13.489720 16079 heartbeater.cc:464] Registering TS with master...
I20260926 04:01:13.489928 16079 heartbeater.cc:511] Master 127.0.98.62:32815 requested a full tablet report, sending...
I20260926 04:01:13.490306 15754 ts_manager.cc:194] Registered new tserver with Master: c3b09a5fb725437ca6c07e2fedf6248d (127.0.98.2:41269)
I20260926 04:01:13.490682 15754 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43569
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 04:01:13.514096 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32815
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:01:13.596314 16084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:01:13.596515 16084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:01:13.596539 16084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:01:13.597747 16084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:01:13.597796 16084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:01:13.597823 16084 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 04:01:13.599306 16084 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16084
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:01:13.599551 16084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:01:13.599785 16084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:01:13.604857 16092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:01:13.604811 16089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:01:13.604885 16090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:01:13.605126 16084 server_base.cc:1034] running on GCE node
I20260926 04:01:13.605331 16084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:01:13.605585 16084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:01:13.606761 16084 hybrid_clock.cc:648] HybridClock initialized: now 1790395273606742 us; error 29 us; skew 500 ppm
Sep 26 04:01:13 dist-test-slave-nhpw krb5kdc[15709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395273, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:01:13.609298 16084 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 04:01:13.609808 16084 webserver.cc:460] Webserver started at http://127.0.98.3:44567/ using document root <none> and password file <none>
I20260926 04:01:13.610056 16084 fs_manager.cc:359] Metadata directory not provided
I20260926 04:01:13.610126 16084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:01:13.610280 16084 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:01:13.611419 16084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-2/data/instance:
uuid: "42019c7843344a1983c3fa631a898039"
format_stamp: "Formatted at 2026-09-26 04:01:13 on dist-test-slave-nhpw"
server_key: "11ce221d825db0c8fbd814c6e762cf7f"
server_key_iv: "9c6fbb99032088ef988deededac6b6eb"
server_key_version: "encryptionkey@0"
I20260926 04:01:13.611852 16084 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-2/wal/instance:
uuid: "42019c7843344a1983c3fa631a898039"
format_stamp: "Formatted at 2026-09-26 04:01:13 on dist-test-slave-nhpw"
server_key: "11ce221d825db0c8fbd814c6e762cf7f"
server_key_iv: "9c6fbb99032088ef988deededac6b6eb"
server_key_version: "encryptionkey@0"
I20260926 04:01:13.613457 16084 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 04:01:13.614262 16099 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:01:13.614446 16084 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 04:01:13.614542 16084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-2/wal
uuid: "42019c7843344a1983c3fa631a898039"
format_stamp: "Formatted at 2026-09-26 04:01:13 on dist-test-slave-nhpw"
server_key: "11ce221d825db0c8fbd814c6e762cf7f"
server_key_iv: "9c6fbb99032088ef988deededac6b6eb"
server_key_version: "encryptionkey@0"
I20260926 04:01:13.614630 16084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:01:13.645644 16084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:01:13.647708 16084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:01:13.647886 16084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:01:13.648137 16084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:01:13.648454 16084 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:01:13.648511 16084 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:01:13.648561 16084 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:01:13.648586 16084 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:01:13.654886 16084 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44153
I20260926 04:01:13.654927 16212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44153 every 8 connection(s)
I20260926 04:01:13.655260 16084 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
I20260926 04:01:13.659269 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 16084
I20260926 04:01:13.659399 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-2/wal/instance
I20260926 04:01:13.659619 392 external_mini_cluster.cc:1428] Setting key 3be40837a8779ae2d1f23eeccd48e555
Sep 26 04:01:13 dist-test-slave-nhpw krb5kdc[15709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395273, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 04:01:13.664970 15794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:13.656059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33433 (local address 127.0.98.62:32815)
0926 04:01:13.656141 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0926 04:01:13.656143 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:01:13.656311 (+ 168us) server_negotiation.cc:407] Connection header received
0926 04:01:13.656874 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:01:13.656875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:01:13.656885 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:01:13.656938 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:01:13.657877 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:01:13.658372 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:01:13.659411 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:01:13.659669 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:01:13.659741 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:01:13.662176 (+ 2435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:01:13.662178 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:01:13.662178 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:01:13.662185 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:01:13.663673 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:01:13.664033 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:01:13.664034 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:01:13.664034 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:01:13.664087 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:01:13.664354 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:01:13.664354 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:01:13.664355 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:01:13.664466 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:01:13.664522 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 04:01:13.664830 (+ 308us) server_negotiation.cc:299] Negotiation successful
0926 04:01:13.664905 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 04:01:13.665179 16215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:13.656009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32815 (local address 127.0.98.3:33433)
0926 04:01:13.656185 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 04:01:13.656221 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 04:01:13.656777 (+ 556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:01:13.657001 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:01:13.657004 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:01:13.657572 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:01:13.657759 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:01:13.657763 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:01:13.658462 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:01:13.658463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:01:13.659298 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:01:13.659300 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:01:13.659750 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:01:13.659751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:01:13.659900 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:01:13.660594 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:01:13.660605 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:01:13.662093 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:01:13.663743 (+ 1650us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:01:13.663749 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:01:13.663750 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:01:13.663967 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:01:13.664175 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:01:13.664176 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:01:13.664176 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:01:13.664282 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:01:13.664541 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:01:13.664545 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:01:13.664725 (+ 180us) client_negotiation.cc:769] Sending connection context
0926 04:01:13.664811 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 04:01:13.664934 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":94,"threads_started":1}
I20260926 04:01:13.665684 16213 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32815
I20260926 04:01:13.665786 16213 heartbeater.cc:464] Registering TS with master...
I20260926 04:01:13.665956 16213 heartbeater.cc:511] Master 127.0.98.62:32815 requested a full tablet report, sending...
I20260926 04:01:13.666380 15754 ts_manager.cc:194] Registered new tserver with Master: 42019c7843344a1983c3fa631a898039 (127.0.98.3:44153)
I20260926 04:01:13.666718 15754 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33433
I20260926 04:01:13.671872 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 04:01:13.679124 15794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:13.673302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40370 (local address 127.0.98.62:32815)
0926 04:01:13.673386 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0926 04:01:13.673388 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:01:13.673498 (+ 110us) server_negotiation.cc:407] Connection header received
0926 04:01:13.673543 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:01:13.673543 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:01:13.673552 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:01:13.673600 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:01:13.674114 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:01:13.674593 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:01:13.675291 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:01:13.675477 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:01:13.675534 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:01:13.676615 (+ 1081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:01:13.676616 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:01:13.676616 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:01:13.676622 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:01:13.678063 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:01:13.678364 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:01:13.678365 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:01:13.678365 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:01:13.678416 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:01:13.678618 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:01:13.678620 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:01:13.678620 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:01:13.678779 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:01:13.678871 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 04:01:13.678954 (+ 83us) server_negotiation.cc:299] Negotiation successful
0926 04:01:13.679015 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 04:01:13.680595 15754 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40370:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 04:01:13.682214 15754 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 04:01:13.688845 16228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:13.684817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34080 (local address 127.0.98.2:41269)
0926 04:01:13.685235 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0926 04:01:13.685239 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 04:01:13.685249 (+ 10us) server_negotiation.cc:407] Connection header received
0926 04:01:13.685285 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:01:13.685287 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:01:13.685303 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:01:13.685598 (+ 295us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:01:13.685651 (+ 53us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:01:13.686295 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:01:13.687915 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:01:13.688412 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:01:13.688507 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:01:13.688571 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 04:01:13.688713 (+ 142us) server_negotiation.cc:299] Negotiation successful
0926 04:01:13.688759 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":88,"threads_started":1}
I20260926 04:01:13.688848 16224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:13.684512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39669 (local address 127.0.0.1:46352)
0926 04:01:13.684623 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0926 04:01:13.684683 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 04:01:13.684750 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:01:13.685385 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:01:13.685386 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:01:13.685388 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:01:13.685430 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:01:13.685431 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:01:13.686747 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:01:13.686748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:01:13.687627 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:01:13.687630 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:01:13.688561 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:01:13.688562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:01:13.688666 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:01:13.688670 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 04:01:13.688703 (+ 33us) client_negotiation.cc:240] Negotiation successful
0926 04:01:13.688769 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"mutex_wait_us":20,"thread_start_us":65,"threads_started":1}
I20260926 04:01:13.688953 16225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:13.684541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44153 (local address 127.0.0.1:40986)
0926 04:01:13.684857 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0926 04:01:13.684867 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 04:01:13.684904 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:01:13.685133 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:01:13.685137 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:01:13.685140 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:01:13.685207 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:01:13.685210 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:01:13.686694 (+ 1484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:01:13.686695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:01:13.687791 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:01:13.687793 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:01:13.688726 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:01:13.688727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:01:13.688786 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:01:13.688790 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 04:01:13.688839 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 04:01:13.688881 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":81,"threads_started":1}
I20260926 04:01:13.688964 16229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:13.685025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46352 (local address 127.0.98.1:39669)
0926 04:01:13.685212 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 04:01:13.685217 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 04:01:13.685227 (+ 10us) server_negotiation.cc:407] Connection header received
0926 04:01:13.685266 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:01:13.685268 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:01:13.685284 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:01:13.685504 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:01:13.685576 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:01:13.686632 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:01:13.687714 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:01:13.688454 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:01:13.688750 (+ 296us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:01:13.688814 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 04:01:13.688839 (+ 25us) server_negotiation.cc:299] Negotiation successful
0926 04:01:13.688881 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":92,"threads_started":1}
I20260926 04:01:13.689215 16227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:13.684768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40986 (local address 127.0.98.3:44153)
0926 04:01:13.684936 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 04:01:13.684941 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 04:01:13.684951 (+ 10us) server_negotiation.cc:407] Connection header received
0926 04:01:13.684993 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:01:13.684995 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:01:13.685013 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:01:13.685101 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:01:13.685303 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:01:13.686577 (+ 1274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:01:13.687878 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:01:13.688558 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:01:13.688616 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:01:13.688695 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 04:01:13.689122 (+ 427us) server_negotiation.cc:299] Negotiation successful
0926 04:01:13.689157 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":90,"threads_started":1}
I20260926 04:01:13.689347 16226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:13.684506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41269 (local address 127.0.0.1:34080)
0926 04:01:13.684699 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 04:01:13.684709 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 04:01:13.684760 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:01:13.685467 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:01:13.685468 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:01:13.685471 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:01:13.685519 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:01:13.685521 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:01:13.686410 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:01:13.686411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:01:13.687831 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:01:13.687833 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:01:13.688507 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:01:13.688508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:01:13.688585 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:01:13.688599 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 04:01:13.688698 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 04:01:13.689284 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":48,"thread_start_us":88,"threads_started":1}
I20260926 04:01:13.689743 15879 tablet_service.cc:1467] Processing CreateTablet for tablet 7a2693d7588f4d27b4d0eb07114051f2 (DEFAULT_TABLE table=test-table [id=7165af1c8e58432a8c3be8ff5eca27d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:01:13.689875 16147 tablet_service.cc:1467] Processing CreateTablet for tablet 7a2693d7588f4d27b4d0eb07114051f2 (DEFAULT_TABLE table=test-table [id=7165af1c8e58432a8c3be8ff5eca27d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:01:13.690124 15879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a2693d7588f4d27b4d0eb07114051f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:01:13.690142 16147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a2693d7588f4d27b4d0eb07114051f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:01:13.690200 16013 tablet_service.cc:1467] Processing CreateTablet for tablet 7a2693d7588f4d27b4d0eb07114051f2 (DEFAULT_TABLE table=test-table [id=7165af1c8e58432a8c3be8ff5eca27d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:01:13.690548 16013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a2693d7588f4d27b4d0eb07114051f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:01:13.693025 16230 tablet_bootstrap.cc:492] T 7a2693d7588f4d27b4d0eb07114051f2 P c3b09a5fb725437ca6c07e2fedf6248d: Bootstrap starting.
I20260926 04:01:13.693397 16231 tablet_bootstrap.cc:492] T 7a2693d7588f4d27b4d0eb07114051f2 P 42019c7843344a1983c3fa631a898039: Bootstrap starting.
I20260926 04:01:13.693607 16232 tablet_bootstrap.cc:492] T 7a2693d7588f4d27b4d0eb07114051f2 P ffccccc56a0949a29ec497d810c39c43: Bootstrap starting.
I20260926 04:01:13.693779 16230 tablet_bootstrap.cc:654] T 7a2693d7588f4d27b4d0eb07114051f2 P c3b09a5fb725437ca6c07e2fedf6248d: Neither blocks nor log segments found. Creating new log.
I20260926 04:01:13.694389 16232 tablet_bootstrap.cc:654] T 7a2693d7588f4d27b4d0eb07114051f2 P ffccccc56a0949a29ec497d810c39c43: Neither blocks nor log segments found. Creating new log.
I20260926 04:01:13.694396 16230 log.cc:824] T 7a2693d7588f4d27b4d0eb07114051f2 P c3b09a5fb725437ca6c07e2fedf6248d: Log is configured to *not* fsync() on all Append() calls
I20260926 04:01:13.694937 16232 log.cc:824] T 7a2693d7588f4d27b4d0eb07114051f2 P ffccccc56a0949a29ec497d810c39c43: Log is configured to *not* fsync() on all Append() calls
I20260926 04:01:13.695178 16230 tablet_bootstrap.cc:492] T 7a2693d7588f4d27b4d0eb07114051f2 P c3b09a5fb725437ca6c07e2fedf6248d: No bootstrap required, opened a new log
I20260926 04:01:13.695252 16230 ts_tablet_manager.cc:1397] T 7a2693d7588f4d27b4d0eb07114051f2 P c3b09a5fb725437ca6c07e2fedf6248d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 04:01:13.695621 16232 tablet_bootstrap.cc:492] T 7a2693d7588f4d27b4d0eb07114051f2 P ffccccc56a0949a29ec497d810c39c43: No bootstrap required, opened a new log
I20260926 04:01:13.695709 16232 ts_tablet_manager.cc:1397] T 7a2693d7588f4d27b4d0eb07114051f2 P ffccccc56a0949a29ec497d810c39c43: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 04:01:13.695776 16231 tablet_bootstrap.cc:654] T 7a2693d7588f4d27b4d0eb07114051f2 P 42019c7843344a1983c3fa631a898039: Neither blocks nor log segments found. Creating new log.
I20260926 04:01:13.696300 16231 log.cc:824] T 7a2693d7588f4d27b4d0eb07114051f2 P 42019c7843344a1983c3fa631a898039: Log is configured to *not* fsync() on all Append() calls
I20260926 04:01:13.697254 16230 raft_consensus.cc:348] T 7a2693d7588f4d27b4d0eb07114051f2 P c3b09a5fb725437ca6c07e2fedf6248d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b09a5fb725437ca6c07e2fedf6248d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41269 } } peers { permanent_uuid: "ffccccc56a0949a29ec497d810c39c43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39669 } } peers { permanent_uuid: "42019c7843344a1983c3fa631a898039" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44153 } }
I20260926 04:01:13.697444 16230 raft_consensus.cc:374] T 7a2693d7588f4d27b4d0eb07114051f2 P c3b09a5fb725437ca6c07e2fedf6248d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:01:13.697491 16230 raft_consensus.cc:729] T 7a2693d7588f4d27b4d0eb07114051f2 P c3b09a5fb725437ca6c07e2fedf6248d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b09a5fb725437ca6c07e2fedf6248d, State: Initialized, Role: FOLLOWER
I20260926 04:01:13.697606 16230 consensus_queue.cc:260] T 7a2693d7588f4d27b4d0eb07114051f2 P c3b09a5fb725437ca6c07e2fedf6248d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b09a5fb725437ca6c07e2fedf6248d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41269 } } peers { permanent_uuid: "ffccccc56a0949a29ec497d810c39c43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39669 } } peers { permanent_uuid: "42019c7843344a1983c3fa631a898039" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44153 } }
I20260926 04:01:13.697525 16232 raft_consensus.cc:348] T 7a2693d7588f4d27b4d0eb07114051f2 P ffccccc56a0949a29ec497d810c39c43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b09a5fb725437ca6c07e2fedf6248d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41269 } } peers { permanent_uuid: "ffccccc56a0949a29ec497d810c39c43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39669 } } peers { permanent_uuid: "42019c7843344a1983c3fa631a898039" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44153 } }
I20260926 04:01:13.697685 16232 raft_consensus.cc:374] T 7a2693d7588f4d27b4d0eb07114051f2 P ffccccc56a0949a29ec497d810c39c43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:01:13.697710 16232 raft_consensus.cc:729] T 7a2693d7588f4d27b4d0eb07114051f2 P ffccccc56a0949a29ec497d810c39c43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffccccc56a0949a29ec497d810c39c43, State: Initialized, Role: FOLLOWER
I20260926 04:01:13.697918 16230 ts_tablet_manager.cc:1428] T 7a2693d7588f4d27b4d0eb07114051f2 P c3b09a5fb725437ca6c07e2fedf6248d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 04:01:13.697866 16232 consensus_queue.cc:260] T 7a2693d7588f4d27b4d0eb07114051f2 P ffccccc56a0949a29ec497d810c39c43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b09a5fb725437ca6c07e2fedf6248d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41269 } } peers { permanent_uuid: "ffccccc56a0949a29ec497d810c39c43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39669 } } peers { permanent_uuid: "42019c7843344a1983c3fa631a898039" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44153 } }
I20260926 04:01:13.697937 16079 heartbeater.cc:503] Master 127.0.98.62:32815 was elected leader, sending a full tablet report...
I20260926 04:01:13.698182 16232 ts_tablet_manager.cc:1428] T 7a2693d7588f4d27b4d0eb07114051f2 P ffccccc56a0949a29ec497d810c39c43: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 04:01:13.698215 15945 heartbeater.cc:503] Master 127.0.98.62:32815 was elected leader, sending a full tablet report...
I20260926 04:01:13.698554 16231 tablet_bootstrap.cc:492] T 7a2693d7588f4d27b4d0eb07114051f2 P 42019c7843344a1983c3fa631a898039: No bootstrap required, opened a new log
I20260926 04:01:13.698626 16231 ts_tablet_manager.cc:1397] T 7a2693d7588f4d27b4d0eb07114051f2 P 42019c7843344a1983c3fa631a898039: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260926 04:01:13.700448 16231 raft_consensus.cc:348] T 7a2693d7588f4d27b4d0eb07114051f2 P 42019c7843344a1983c3fa631a898039 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b09a5fb725437ca6c07e2fedf6248d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41269 } } peers { permanent_uuid: "ffccccc56a0949a29ec497d810c39c43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39669 } } peers { permanent_uuid: "42019c7843344a1983c3fa631a898039" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44153 } }
I20260926 04:01:13.700585 16231 raft_consensus.cc:374] T 7a2693d7588f4d27b4d0eb07114051f2 P 42019c7843344a1983c3fa631a898039 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:01:13.700608 16231 raft_consensus.cc:729] T 7a2693d7588f4d27b4d0eb07114051f2 P 42019c7843344a1983c3fa631a898039 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42019c7843344a1983c3fa631a898039, State: Initialized, Role: FOLLOWER
I20260926 04:01:13.700774 16231 consensus_queue.cc:260] T 7a2693d7588f4d27b4d0eb07114051f2 P 42019c7843344a1983c3fa631a898039 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b09a5fb725437ca6c07e2fedf6248d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41269 } } peers { permanent_uuid: "ffccccc56a0949a29ec497d810c39c43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39669 } } peers { permanent_uuid: "42019c7843344a1983c3fa631a898039" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44153 } }
I20260926 04:01:13.701053 16231 ts_tablet_manager.cc:1428] T 7a2693d7588f4d27b4d0eb07114051f2 P 42019c7843344a1983c3fa631a898039: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 04:01:13.701176 16213 heartbeater.cc:503] Master 127.0.98.62:32815 was elected leader, sending a full tablet report...
W20260926 04:01:13.729722 16080 tablet.cc:2369] T 7a2693d7588f4d27b4d0eb07114051f2 P c3b09a5fb725437ca6c07e2fedf6248d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 04:01:13.774463 16237 raft_consensus.cc:482] T 7a2693d7588f4d27b4d0eb07114051f2 P ffccccc56a0949a29ec497d810c39c43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 04:01:13.774593 16237 raft_consensus.cc:504] T 7a2693d7588f4d27b4d0eb07114051f2 P ffccccc56a0949a29ec497d810c39c43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b09a5fb725437ca6c07e2fedf6248d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41269 } } peers { permanent_uuid: "ffccccc56a0949a29ec497d810c39c43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39669 } } peers { permanent_uuid: "42019c7843344a1983c3fa631a898039" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44153 } }
I20260926 04:01:13.775048 16237 leader_election.cc:288] T 7a2693d7588f4d27b4d0eb07114051f2 P ffccccc56a0949a29ec497d810c39c43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c3b09a5fb725437ca6c07e2fedf6248d (127.0.98.2:41269), 42019c7843344a1983c3fa631a898039 (127.0.98.3:44153)
I20260926 04:01:13.778218 15947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:13.775243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44153 (local address 127.0.98.1:59371)
0926 04:01:13.775312 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0926 04:01:13.775326 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 04:01:13.775379 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:01:13.775525 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:01:13.775526 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:01:13.775527 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:01:13.775605 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:01:13.775607 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:01:13.776403 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:01:13.776404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:01:13.777301 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:01:13.777304 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:01:13.777934 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:01:13.777935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:01:13.778025 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:01:13.778029 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 04:01:13.778091 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 04:01:13.778136 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":3}
I20260926 04:01:13.778237 16227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:13.775317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59371 (local address 127.0.98.3:44153)
0926 04:01:13.775394 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 04:01:13.775396 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:01:13.775405 (+ 9us) server_negotiation.cc:407] Connection header received
0926 04:01:13.775433 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:01:13.775433 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:01:13.775443 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:01:13.775494 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:01:13.775698 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:01:13.776308 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:01:13.777394 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:01:13.777851 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:01:13.777914 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:01:13.777923 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 04:01:13.778123 (+ 200us) server_negotiation.cc:299] Negotiation successful
0926 04:01:13.778164 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 04:01:13.778472 16228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:13.775306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53945 (local address 127.0.98.2:41269)
0926 04:01:13.775356 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 04:01:13.775357 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 04:01:13.775738 (+ 381us) server_negotiation.cc:407] Connection header received
0926 04:01:13.775776 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:01:13.775776 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:01:13.775788 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:01:13.775825 (+ 37us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:01:13.776025 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:01:13.776587 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:01:13.777630 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:01:13.778071 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:01:13.778133 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:01:13.778143 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 04:01:13.778350 (+ 207us) server_negotiation.cc:299] Negotiation successful
0926 04:01:13.778398 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 04:01:13.778586 16167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2693d7588f4d27b4d0eb07114051f2" candidate_uuid: "ffccccc56a0949a29ec497d810c39c43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42019c7843344a1983c3fa631a898039" is_pre_election: true
I20260926 04:01:13.778707 16239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:13.775246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41269 (local address 127.0.98.1:53945)
0926 04:01:13.775634 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0926 04:01:13.775643 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 04:01:13.775724 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:01:13.775852 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:01:13.775853 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:01:13.775855 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:01:13.775920 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:01:13.775921 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:01:13.776703 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:01:13.776704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:01:13.777543 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:01:13.777546 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:01:13.778153 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:01:13.778154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:01:13.778233 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:01:13.778236 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 04:01:13.778576 (+ 340us) client_negotiation.cc:240] Negotiation successful
0926 04:01:13.778626 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":367,"mutex_wait_us":124,"thread_start_us":156,"threads_started":1}
I20260926 04:01:13.778784 16167 raft_consensus.cc:2454] T 7a2693d7588f4d27b4d0eb07114051f2 P 42019c7843344a1983c3fa631a898039 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ffccccc56a0949a29ec497d810c39c43 in term 0.
I20260926 04:01:13.779024 15835 leader_election.cc:302] T 7a2693d7588f4d27b4d0eb07114051f2 P ffccccc56a0949a29ec497d810c39c43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42019c7843344a1983c3fa631a898039, ffccccc56a0949a29ec497d810c39c43; no voters:
I20260926 04:01:13.779048 16033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2693d7588f4d27b4d0eb07114051f2" candidate_uuid: "ffccccc56a0949a29ec497d810c39c43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b09a5fb725437ca6c07e2fedf6248d" is_pre_election: true
I20260926 04:01:13.779153 16237 raft_consensus.cc:2781] T 7a2693d7588f4d27b4d0eb07114051f2 P ffccccc56a0949a29ec497d810c39c43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 04:01:13.779214 16237 raft_consensus.cc:482] T 7a2693d7588f4d27b4d0eb07114051f2 P ffccccc56a0949a29ec497d810c39c43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 04:01:13.779206 16033 raft_consensus.cc:2454] T 7a2693d7588f4d27b4d0eb07114051f2 P c3b09a5fb725437ca6c07e2fedf6248d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ffccccc56a0949a29ec497d810c39c43 in term 0.
I20260926 04:01:13.779254 16237 raft_consensus.cc:3037] T 7a2693d7588f4d27b4d0eb07114051f2 P ffccccc56a0949a29ec497d810c39c43 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:01:13.779964 16237 raft_consensus.cc:504] T 7a2693d7588f4d27b4d0eb07114051f2 P ffccccc56a0949a29ec497d810c39c43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b09a5fb725437ca6c07e2fedf6248d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41269 } } peers { permanent_uuid: "ffccccc56a0949a29ec497d810c39c43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39669 } } peers { permanent_uuid: "42019c7843344a1983c3fa631a898039" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44153 } }
I20260926 04:01:13.780122 16237 leader_election.cc:288] T 7a2693d7588f4d27b4d0eb07114051f2 P ffccccc56a0949a29ec497d810c39c43 [CANDIDATE]: Term 1 election: Requested vote from peers c3b09a5fb725437ca6c07e2fedf6248d (127.0.98.2:41269), 42019c7843344a1983c3fa631a898039 (127.0.98.3:44153)
I20260926 04:01:13.780283 16033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2693d7588f4d27b4d0eb07114051f2" candidate_uuid: "ffccccc56a0949a29ec497d810c39c43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c3b09a5fb725437ca6c07e2fedf6248d"
I20260926 04:01:13.780373 16167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a2693d7588f4d27b4d0eb07114051f2" candidate_uuid: "ffccccc56a0949a29ec497d810c39c43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42019c7843344a1983c3fa631a898039"
I20260926 04:01:13.780391 16033 raft_consensus.cc:3037] T 7a2693d7588f4d27b4d0eb07114051f2 P c3b09a5fb725437ca6c07e2fedf6248d [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:01:13.780457 16167 raft_consensus.cc:3037] T 7a2693d7588f4d27b4d0eb07114051f2 P 42019c7843344a1983c3fa631a898039 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:01:13.781198 16033 raft_consensus.cc:2454] T 7a2693d7588f4d27b4d0eb07114051f2 P c3b09a5fb725437ca6c07e2fedf6248d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffccccc56a0949a29ec497d810c39c43 in term 1.
I20260926 04:01:13.781211 16167 raft_consensus.cc:2454] T 7a2693d7588f4d27b4d0eb07114051f2 P 42019c7843344a1983c3fa631a898039 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ffccccc56a0949a29ec497d810c39c43 in term 1.
I20260926 04:01:13.781379 15834 leader_election.cc:302] T 7a2693d7588f4d27b4d0eb07114051f2 P ffccccc56a0949a29ec497d810c39c43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c3b09a5fb725437ca6c07e2fedf6248d, ffccccc56a0949a29ec497d810c39c43; no voters:
I20260926 04:01:13.781476 16237 raft_consensus.cc:2781] T 7a2693d7588f4d27b4d0eb07114051f2 P ffccccc56a0949a29ec497d810c39c43 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:01:13.781630 16237 raft_consensus.cc:686] T 7a2693d7588f4d27b4d0eb07114051f2 P ffccccc56a0949a29ec497d810c39c43 [term 1 LEADER]: Becoming Leader. State: Replica: ffccccc56a0949a29ec497d810c39c43, State: Running, Role: LEADER
I20260926 04:01:13.781721 16237 consensus_queue.cc:237] T 7a2693d7588f4d27b4d0eb07114051f2 P ffccccc56a0949a29ec497d810c39c43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b09a5fb725437ca6c07e2fedf6248d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41269 } } peers { permanent_uuid: "ffccccc56a0949a29ec497d810c39c43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39669 } } peers { permanent_uuid: "42019c7843344a1983c3fa631a898039" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44153 } }
I20260926 04:01:13.782557 15753 catalog_manager.cc:5495] T 7a2693d7588f4d27b4d0eb07114051f2 P ffccccc56a0949a29ec497d810c39c43 reported cstate change: term changed from 0 to 1, leader changed from <none> to ffccccc56a0949a29ec497d810c39c43 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "ffccccc56a0949a29ec497d810c39c43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b09a5fb725437ca6c07e2fedf6248d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ffccccc56a0949a29ec497d810c39c43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39669 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "42019c7843344a1983c3fa631a898039" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44153 } health_report { overall_health: UNKNOWN } } }
I20260926 04:01:13.808226 16229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:13.805753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46358 (local address 127.0.98.1:39669)
0926 04:01:13.805837 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0926 04:01:13.805839 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:01:13.805846 (+ 7us) server_negotiation.cc:407] Connection header received
0926 04:01:13.805891 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:01:13.805891 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:01:13.805910 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:01:13.805991 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 04:01:13.806142 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:01:13.806646 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:01:13.807432 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:01:13.807639 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:01:13.807716 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:01:13.807881 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 04:01:13.807991 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 04:01:13.808078 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 04:01:13.808117 (+ 39us) server_negotiation.cc:299] Negotiation successful
0926 04:01:13.808138 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 04:01:13.810101 16167 raft_consensus.cc:1261] T 7a2693d7588f4d27b4d0eb07114051f2 P 42019c7843344a1983c3fa631a898039 [term 1 FOLLOWER]: Refusing update from remote peer ffccccc56a0949a29ec497d810c39c43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:01:13.810187 16033 raft_consensus.cc:1261] T 7a2693d7588f4d27b4d0eb07114051f2 P c3b09a5fb725437ca6c07e2fedf6248d [term 1 FOLLOWER]: Refusing update from remote peer ffccccc56a0949a29ec497d810c39c43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:01:13.810554 16240 consensus_queue.cc:1035] T 7a2693d7588f4d27b4d0eb07114051f2 P ffccccc56a0949a29ec497d810c39c43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "42019c7843344a1983c3fa631a898039" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:01:13.810688 16237 consensus_queue.cc:1035] T 7a2693d7588f4d27b4d0eb07114051f2 P ffccccc56a0949a29ec497d810c39c43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c3b09a5fb725437ca6c07e2fedf6248d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:01:13.813103 16243 mvcc.cc:205] Tried to move back new op lower bound from 7333459041523253248 to 7333459041410420736. Current Snapshot: MvccSnapshot[applied={T|T < 7333459041523253248}]
I20260926 04:01:13.813537 16244 mvcc.cc:205] Tried to move back new op lower bound from 7333459041523253248 to 7333459041410420736. Current Snapshot: MvccSnapshot[applied={T|T < 7333459041523253248}]
I20260926 04:01:13.815769 15753 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 04:01:13.815869 15753 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 04:01:13.816666 15753 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5ad1f0dd809a452ca0355fe421f1b3ad: Sending DeleteTablet for 3 replicas of tablet 7a2693d7588f4d27b4d0eb07114051f2
I20260926 04:01:13.817054 16147 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a2693d7588f4d27b4d0eb07114051f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:01:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40986
I20260926 04:01:13.817039 16013 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a2693d7588f4d27b4d0eb07114051f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:01:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34080
I20260926 04:01:13.817266 15879 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a2693d7588f4d27b4d0eb07114051f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:01:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46352
I20260926 04:01:13.817504 16255 tablet_replica.cc:330] T 7a2693d7588f4d27b4d0eb07114051f2 P ffccccc56a0949a29ec497d810c39c43: stopping tablet replica
I20260926 04:01:13.817582 16253 tablet_replica.cc:330] T 7a2693d7588f4d27b4d0eb07114051f2 P c3b09a5fb725437ca6c07e2fedf6248d: stopping tablet replica
I20260926 04:01:13.817605 16255 raft_consensus.cc:2229] T 7a2693d7588f4d27b4d0eb07114051f2 P ffccccc56a0949a29ec497d810c39c43 [term 1 LEADER]: Raft consensus shutting down.
I20260926 04:01:13.817592 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 15816
I20260926 04:01:13.817655 16253 raft_consensus.cc:2229] T 7a2693d7588f4d27b4d0eb07114051f2 P c3b09a5fb725437ca6c07e2fedf6248d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 04:01:13.818683 16253 raft_consensus.cc:2258] T 7a2693d7588f4d27b4d0eb07114051f2 P c3b09a5fb725437ca6c07e2fedf6248d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:01:13.818884 16254 tablet_replica.cc:330] T 7a2693d7588f4d27b4d0eb07114051f2 P 42019c7843344a1983c3fa631a898039: stopping tablet replica
I20260926 04:01:13.818979 16254 raft_consensus.cc:2229] T 7a2693d7588f4d27b4d0eb07114051f2 P 42019c7843344a1983c3fa631a898039 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 04:01:13.819072 16254 raft_consensus.cc:2258] T 7a2693d7588f4d27b4d0eb07114051f2 P 42019c7843344a1983c3fa631a898039 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:01:13.819168 16253 ts_tablet_manager.cc:1905] T 7a2693d7588f4d27b4d0eb07114051f2 P c3b09a5fb725437ca6c07e2fedf6248d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:01:13.819495 16254 ts_tablet_manager.cc:1905] T 7a2693d7588f4d27b4d0eb07114051f2 P 42019c7843344a1983c3fa631a898039: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:01:13.820784 16253 ts_tablet_manager.cc:1918] T 7a2693d7588f4d27b4d0eb07114051f2 P c3b09a5fb725437ca6c07e2fedf6248d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 04:01:13.820871 16253 log.cc:1196] T 7a2693d7588f4d27b4d0eb07114051f2 P c3b09a5fb725437ca6c07e2fedf6248d: Deleting WAL directory at /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-1/wal/wals/7a2693d7588f4d27b4d0eb07114051f2
I20260926 04:01:13.821147 16253 ts_tablet_manager.cc:1939] T 7a2693d7588f4d27b4d0eb07114051f2 P c3b09a5fb725437ca6c07e2fedf6248d: Deleting consensus metadata
I20260926 04:01:13.821431 16254 ts_tablet_manager.cc:1918] T 7a2693d7588f4d27b4d0eb07114051f2 P 42019c7843344a1983c3fa631a898039: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 04:01:13.821501 16254 log.cc:1196] T 7a2693d7588f4d27b4d0eb07114051f2 P 42019c7843344a1983c3fa631a898039: Deleting WAL directory at /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-2/wal/wals/7a2693d7588f4d27b4d0eb07114051f2
I20260926 04:01:13.821576 15740 catalog_manager.cc:4841] TS c3b09a5fb725437ca6c07e2fedf6248d (127.0.98.2:41269): tablet 7a2693d7588f4d27b4d0eb07114051f2 (table test-table [id=7165af1c8e58432a8c3be8ff5eca27d1]) successfully deleted
I20260926 04:01:13.821753 16254 ts_tablet_manager.cc:1939] T 7a2693d7588f4d27b4d0eb07114051f2 P 42019c7843344a1983c3fa631a898039: Deleting consensus metadata
I20260926 04:01:13.822420 15741 catalog_manager.cc:4841] TS 42019c7843344a1983c3fa631a898039 (127.0.98.3:44153): tablet 7a2693d7588f4d27b4d0eb07114051f2 (table test-table [id=7165af1c8e58432a8c3be8ff5eca27d1]) successfully deleted
W20260926 04:01:13.824050 15739 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39669 (error 108)
I20260926 04:01:13.824325 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 15950
I20260926 04:01:13.824616 16226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:13.824499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39669 (local address 127.0.0.1:46368)
0926 04:01:13.824545 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 04:01:13.824576 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260926 04:01:13.824910 15739 catalog_manager.cc:4584] TS ffccccc56a0949a29ec497d810c39c43 (127.0.98.1:39669): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7a2693d7588f4d27b4d0eb07114051f2: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39669: connect: Connection refused (error 111)
I20260926 04:01:13.829186 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 16084
I20260926 04:01:13.833395 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 15722
I20260926 04:01:33.838598 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32815
--webserver_interface=127.0.98.62
--webserver_port=40033
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32815
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:01:33.918479 16257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:01:33.918715 16257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:01:33.918754 16257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:01:33.919878 16257 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 04:01:33.919942 16257 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 04:01:33.919965 16257 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 04:01:33.919992 16257 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:01:33.920025 16257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:01:33.920058 16257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:01:33.920086 16257 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:01:33.920112 16257 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:01:33.921562 16257 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32815
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=40033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:01:33.921828 16257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:01:33.922046 16257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:01:33.930500 16266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:01:33.930500 16264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:01:33.930557 16263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:01:33.930546 16257 server_base.cc:1034] running on GCE node
I20260926 04:01:33.930968 16257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:01:33.931289 16257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:01:33.932451 16257 hybrid_clock.cc:648] HybridClock initialized: now 1790395293932414 us; error 52 us; skew 500 ppm
Sep 26 04:01:33 dist-test-slave-nhpw krb5kdc[15709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:01:33.935106 16257 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 04:01:33.935580 16257 webserver.cc:460] Webserver started at http://127.0.98.62:40033/ using document root <none> and password file <none>
I20260926 04:01:33.935829 16257 fs_manager.cc:359] Metadata directory not provided
I20260926 04:01:33.935913 16257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:01:33.937403 16257 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260926 04:01:33.938138 16273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:01:33.938351 16257 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 04:01:33.938448 16257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/master-0/wal
uuid: "5ad1f0dd809a452ca0355fe421f1b3ad"
format_stamp: "Formatted at 2026-09-26 04:01:13 on dist-test-slave-nhpw"
server_key: "e81730a0b3966cb0eb8ead6f01601210"
server_key_iv: "08b0aaa8da8b10f6beb5f6eca5a53cf2"
server_key_version: "encryptionkey@0"
I20260926 04:01:33.938894 16257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:01:33.955858 16257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:01:33.958290 16257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:01:33.958460 16257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:01:33.962736 16257 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32815
I20260926 04:01:33.962771 16325 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32815 every 8 connection(s)
I20260926 04:01:33.963155 16257 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
I20260926 04:01:33.963562 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 16257
I20260926 04:01:33.963883 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:39669
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:32815
--webserver_port=38119
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 04:01:33.964892 16326 sys_catalog.cc:263] Verifying existing consensus state
I20260926 04:01:33.965556 16326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ad1f0dd809a452ca0355fe421f1b3ad: Bootstrap starting.
I20260926 04:01:33.974217 16326 log.cc:824] T 00000000000000000000000000000000 P 5ad1f0dd809a452ca0355fe421f1b3ad: Log is configured to *not* fsync() on all Append() calls
I20260926 04:01:33.976161 16326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ad1f0dd809a452ca0355fe421f1b3ad: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 04:01:33.976464 16326 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ad1f0dd809a452ca0355fe421f1b3ad: Bootstrap complete.
I20260926 04:01:33.978847 16326 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ad1f0dd809a452ca0355fe421f1b3ad [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ad1f0dd809a452ca0355fe421f1b3ad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32815 } }
I20260926 04:01:33.979113 16326 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ad1f0dd809a452ca0355fe421f1b3ad [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ad1f0dd809a452ca0355fe421f1b3ad, State: Initialized, Role: FOLLOWER
I20260926 04:01:33.979306 16326 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ad1f0dd809a452ca0355fe421f1b3ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ad1f0dd809a452ca0355fe421f1b3ad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32815 } }
I20260926 04:01:33.979388 16326 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ad1f0dd809a452ca0355fe421f1b3ad [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:01:33.979445 16326 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ad1f0dd809a452ca0355fe421f1b3ad [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:01:33.979494 16326 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ad1f0dd809a452ca0355fe421f1b3ad [term 1 FOLLOWER]: Advancing to term 2
I20260926 04:01:33.980481 16326 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ad1f0dd809a452ca0355fe421f1b3ad [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ad1f0dd809a452ca0355fe421f1b3ad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32815 } }
I20260926 04:01:33.980602 16326 leader_election.cc:302] T 00000000000000000000000000000000 P 5ad1f0dd809a452ca0355fe421f1b3ad [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ad1f0dd809a452ca0355fe421f1b3ad; no voters:
I20260926 04:01:33.980790 16326 leader_election.cc:288] T 00000000000000000000000000000000 P 5ad1f0dd809a452ca0355fe421f1b3ad [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 04:01:33.980825 16330 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ad1f0dd809a452ca0355fe421f1b3ad [term 2 FOLLOWER]: Leader election won for term 2
I20260926 04:01:33.980947 16330 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ad1f0dd809a452ca0355fe421f1b3ad [term 2 LEADER]: Becoming Leader. State: Replica: 5ad1f0dd809a452ca0355fe421f1b3ad, State: Running, Role: LEADER
I20260926 04:01:33.981056 16326 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ad1f0dd809a452ca0355fe421f1b3ad [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:01:33.981086 16330 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ad1f0dd809a452ca0355fe421f1b3ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ad1f0dd809a452ca0355fe421f1b3ad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32815 } }
I20260926 04:01:33.981361 16330 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ad1f0dd809a452ca0355fe421f1b3ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ad1f0dd809a452ca0355fe421f1b3ad. Latest consensus state: current_term: 2 leader_uuid: "5ad1f0dd809a452ca0355fe421f1b3ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ad1f0dd809a452ca0355fe421f1b3ad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32815 } } }
I20260926 04:01:33.981431 16330 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ad1f0dd809a452ca0355fe421f1b3ad [sys.catalog]: This master's current role is: LEADER
I20260926 04:01:33.981566 16331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ad1f0dd809a452ca0355fe421f1b3ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5ad1f0dd809a452ca0355fe421f1b3ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ad1f0dd809a452ca0355fe421f1b3ad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32815 } } }
I20260926 04:01:33.981681 16331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ad1f0dd809a452ca0355fe421f1b3ad [sys.catalog]: This master's current role is: LEADER
I20260926 04:01:33.981853 16335 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:01:33.982717 16335 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:01:33.982841 16335 catalog_manager.cc:1259] Loaded cluster ID: f52174d07b3f4bb79225777d91d151b2
I20260926 04:01:33.983089 16335 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 04:01:33.983918 16335 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:01:33.984154 16335 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5ad1f0dd809a452ca0355fe421f1b3ad: Loaded TSK: 0
I20260926 04:01:33.991987 16335 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ad1f0dd809a452ca0355fe421f1b3ad: Generated new TSK 1
I20260926 04:01:33.992090 16335 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 04:01:34.057953 16328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:01:34.058215 16328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:01:34.058259 16328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:01:34.059477 16328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:01:34.059557 16328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:01:34.059607 16328 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 04:01:34.061146 16328 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:39669
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=38119
--tserver_master_addrs=127.0.98.62:32815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:01:34.061415 16328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:01:34.061653 16328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:01:34.066931 16354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:01:34.066936 16356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:01:34.067058 16328 server_base.cc:1034] running on GCE node
W20260926 04:01:34.066926 16353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:01:34.067445 16328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:01:34.067745 16328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:01:34.068907 16328 hybrid_clock.cc:648] HybridClock initialized: now 1790395294068887 us; error 34 us; skew 500 ppm
Sep 26 04:01:34 dist-test-slave-nhpw krb5kdc[15709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395294, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:01:34.072088 16328 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 04:01:34.072701 16328 webserver.cc:460] Webserver started at http://127.0.98.1:38119/ using document root <none> and password file <none>
I20260926 04:01:34.072947 16328 fs_manager.cc:359] Metadata directory not provided
I20260926 04:01:34.073035 16328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:01:34.074472 16328 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:01:34.075296 16363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:01:34.075541 16328 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:01:34.075656 16328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-0/wal
uuid: "ffccccc56a0949a29ec497d810c39c43"
format_stamp: "Formatted at 2026-09-26 04:01:13 on dist-test-slave-nhpw"
server_key: "88374dfc5922c3417e129b86cdc294a1"
server_key_iv: "b4eb3d1e4dc1023e48409622dae899da"
server_key_version: "encryptionkey@0"
I20260926 04:01:34.076020 16328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:01:34.106137 16328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:01:34.108600 16328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:01:34.108829 16328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:01:34.109112 16328 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:01:34.109589 16370 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 04:01:34.111543 16328 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 04:01:34.111636 16328 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 04:01:34.111683 16328 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 04:01:34.112344 16328 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 04:01:34.112402 16328 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 04:01:34.112470 16370 tablet_bootstrap.cc:492] T 7a2693d7588f4d27b4d0eb07114051f2 P ffccccc56a0949a29ec497d810c39c43: Bootstrap starting.
I20260926 04:01:34.119874 16328 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39669
I20260926 04:01:34.120261 16328 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
I20260926 04:01:34.120790 16477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39669 every 8 connection(s)
I20260926 04:01:34.123198 16370 log.cc:824] T 7a2693d7588f4d27b4d0eb07114051f2 P ffccccc56a0949a29ec497d810c39c43: Log is configured to *not* fsync() on all Append() calls
I20260926 04:01:34.126082 16370 tablet_bootstrap.cc:492] T 7a2693d7588f4d27b4d0eb07114051f2 P ffccccc56a0949a29ec497d810c39c43: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 04:01:34.126443 16370 tablet_bootstrap.cc:492] T 7a2693d7588f4d27b4d0eb07114051f2 P ffccccc56a0949a29ec497d810c39c43: Bootstrap complete.
Sep 26 04:01:34 dist-test-slave-nhpw krb5kdc[15709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395294, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 04:01:34.127676 16370 ts_tablet_manager.cc:1397] T 7a2693d7588f4d27b4d0eb07114051f2 P ffccccc56a0949a29ec497d810c39c43: Time spent bootstrapping tablet: real 0.015s user 0.012s sys 0.000s
I20260926 04:01:34.128927 16370 raft_consensus.cc:348] T 7a2693d7588f4d27b4d0eb07114051f2 P ffccccc56a0949a29ec497d810c39c43 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b09a5fb725437ca6c07e2fedf6248d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41269 } } peers { permanent_uuid: "ffccccc56a0949a29ec497d810c39c43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39669 } } peers { permanent_uuid: "42019c7843344a1983c3fa631a898039" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44153 } }
I20260926 04:01:34.129233 16370 raft_consensus.cc:729] T 7a2693d7588f4d27b4d0eb07114051f2 P ffccccc56a0949a29ec497d810c39c43 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffccccc56a0949a29ec497d810c39c43, State: Initialized, Role: FOLLOWER
I20260926 04:01:34.129354 16370 consensus_queue.cc:260] T 7a2693d7588f4d27b4d0eb07114051f2 P ffccccc56a0949a29ec497d810c39c43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c3b09a5fb725437ca6c07e2fedf6248d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41269 } } peers { permanent_uuid: "ffccccc56a0949a29ec497d810c39c43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39669 } } peers { permanent_uuid: "42019c7843344a1983c3fa631a898039" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44153 } }
I20260926 04:01:34.129545 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 16328
I20260926 04:01:34.129608 16370 ts_tablet_manager.cc:1428] T 7a2693d7588f4d27b4d0eb07114051f2 P ffccccc56a0949a29ec497d810c39c43: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20260926 04:01:34.129812 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:41269
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:32815
--webserver_port=46101
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 04:01:34.134087 16480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:34.121000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32815 (local address 127.0.98.1:40677)
0926 04:01:34.121325 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0926 04:01:34.121373 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 04:01:34.121809 (+ 436us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:01:34.122319 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:01:34.122323 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:01:34.122623 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:01:34.122812 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:01:34.122817 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:01:34.124211 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:01:34.124212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:01:34.125125 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:01:34.125128 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:01:34.125895 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:01:34.125896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:01:34.126006 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:01:34.126634 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:01:34.126642 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:01:34.128041 (+ 1399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:01:34.132095 (+ 4054us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:01:34.132102 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:01:34.132104 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:01:34.132449 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:01:34.132801 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:01:34.132802 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:01:34.132802 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:01:34.132949 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:01:34.133461 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:01:34.133468 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:01:34.133616 (+ 148us) client_negotiation.cc:769] Sending connection context
0926 04:01:34.133733 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 04:01:34.133894 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":85,"threads_started":1}
I20260926 04:01:34.134385 16481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:34.121085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40677 (local address 127.0.98.62:32815)
0926 04:01:34.121962 (+ 877us) server_negotiation.cc:206] Beginning negotiation
0926 04:01:34.121969 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 04:01:34.122007 (+ 38us) server_negotiation.cc:407] Connection header received
0926 04:01:34.122107 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:01:34.122112 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:01:34.122149 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:01:34.122290 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:01:34.122946 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:01:34.124084 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:01:34.125233 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:01:34.125626 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:01:34.125749 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:01:34.128151 (+ 2402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:01:34.128157 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:01:34.128157 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:01:34.128172 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:01:34.131987 (+ 3815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:01:34.132573 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:01:34.132576 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:01:34.132577 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:01:34.132720 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:01:34.133034 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:01:34.133035 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:01:34.133036 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:01:34.133373 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:01:34.133469 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 04:01:34.133818 (+ 349us) server_negotiation.cc:299] Negotiation successful
0926 04:01:34.133978 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":834,"thread_start_us":75,"threads_started":1}
I20260926 04:01:34.135160 16478 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32815
I20260926 04:01:34.135257 16478 heartbeater.cc:464] Registering TS with master...
I20260926 04:01:34.135417 16478 heartbeater.cc:511] Master 127.0.98.62:32815 requested a full tablet report, sending...
I20260926 04:01:34.136189 16290 ts_manager.cc:194] Registered new tserver with Master: ffccccc56a0949a29ec497d810c39c43 (127.0.98.1:39669)
I20260926 04:01:34.137169 16290 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40677
I20260926 04:01:34.138278 16478 heartbeater.cc:503] Master 127.0.98.62:32815 was elected leader, sending a full tablet report...
Sep 26 04:01:34 dist-test-slave-nhpw krb5kdc[15709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395293, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260926 04:01:34.145223 16486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:34.137118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38276 (local address 127.0.98.1:39669)
0926 04:01:34.137645 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0926 04:01:34.137649 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 04:01:34.137658 (+ 9us) server_negotiation.cc:407] Connection header received
0926 04:01:34.137693 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:01:34.137695 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:01:34.137708 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:01:34.137799 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:01:34.138209 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:01:34.139092 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:01:34.139839 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:01:34.140139 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:01:34.140223 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:01:34.142402 (+ 2179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:01:34.142408 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:01:34.142408 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:01:34.142414 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:01:34.143969 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:01:34.144407 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:01:34.144409 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:01:34.144409 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:01:34.144462 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:01:34.144690 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:01:34.144691 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:01:34.144692 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:01:34.144832 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:01:34.144896 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 04:01:34.145082 (+ 186us) server_negotiation.cc:299] Negotiation successful
0926 04:01:34.145156 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":77,"threads_started":1}
I20260926 04:01:34.145257 16485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:34.136859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39669 (local address 127.0.0.1:38276)
0926 04:01:34.137371 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0926 04:01:34.137406 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 04:01:34.137490 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:01:34.137802 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:01:34.137807 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:01:34.138069 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:01:34.138113 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:01:34.138115 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:01:34.139177 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:01:34.139178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:01:34.139744 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:01:34.139746 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:01:34.140220 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:01:34.140223 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:01:34.140305 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:01:34.140851 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:01:34.140856 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:01:34.142314 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:01:34.144060 (+ 1746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:01:34.144067 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:01:34.144069 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:01:34.144325 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:01:34.144529 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:01:34.144529 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:01:34.144530 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:01:34.144594 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:01:34.144898 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:01:34.144903 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:01:34.144956 (+ 53us) client_negotiation.cc:769] Sending connection context
0926 04:01:34.145041 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 04:01:34.145154 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":91,"threads_started":1}
I20260926 04:01:34.145648 16412 tablet_service.cc:1514] Processing DeleteTablet for tablet 7a2693d7588f4d27b4d0eb07114051f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:01:13 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38276
I20260926 04:01:34.148787 16487 tablet_replica.cc:330] stopping tablet replica
I20260926 04:01:34.148932 16487 raft_consensus.cc:2229] T 7a2693d7588f4d27b4d0eb07114051f2 P ffccccc56a0949a29ec497d810c39c43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 04:01:34.149017 16487 raft_consensus.cc:2258] T 7a2693d7588f4d27b4d0eb07114051f2 P ffccccc56a0949a29ec497d810c39c43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:01:34.149329 16487 ts_tablet_manager.cc:1905] T 7a2693d7588f4d27b4d0eb07114051f2 P ffccccc56a0949a29ec497d810c39c43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:01:34.151561 16487 ts_tablet_manager.cc:1918] T 7a2693d7588f4d27b4d0eb07114051f2 P ffccccc56a0949a29ec497d810c39c43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 04:01:34.151692 16487 log.cc:1196] T 7a2693d7588f4d27b4d0eb07114051f2 P ffccccc56a0949a29ec497d810c39c43: Deleting WAL directory at /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-0/wal/wals/7a2693d7588f4d27b4d0eb07114051f2
I20260926 04:01:34.151937 16487 ts_tablet_manager.cc:1939] T 7a2693d7588f4d27b4d0eb07114051f2 P ffccccc56a0949a29ec497d810c39c43: Deleting consensus metadata
I20260926 04:01:34.152462 16275 catalog_manager.cc:4841] TS ffccccc56a0949a29ec497d810c39c43 (127.0.98.1:39669): tablet 7a2693d7588f4d27b4d0eb07114051f2 (table test-table [id=7165af1c8e58432a8c3be8ff5eca27d1]) successfully deleted
W20260926 04:01:34.213989 16484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:01:34.214278 16484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:01:34.214346 16484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:01:34.215843 16484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:01:34.215884 16484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:01:34.215911 16484 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 04:01:34.217353 16484 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:41269
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=46101
--tserver_master_addrs=127.0.98.62:32815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16484
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:01:34.217600 16484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:01:34.217844 16484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:01:34.222787 16493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:01:34.222913 16484 server_base.cc:1034] running on GCE node
W20260926 04:01:34.222788 16495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:01:34.222811 16492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:01:34.223165 16484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:01:34.223357 16484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:01:34.224493 16484 hybrid_clock.cc:648] HybridClock initialized: now 1790395294224477 us; error 29 us; skew 500 ppm
Sep 26 04:01:34 dist-test-slave-nhpw krb5kdc[15709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395294, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:01:34.226974 16484 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 04:01:34.227461 16484 webserver.cc:460] Webserver started at http://127.0.98.2:46101/ using document root <none> and password file <none>
I20260926 04:01:34.227699 16484 fs_manager.cc:359] Metadata directory not provided
I20260926 04:01:34.227782 16484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:01:34.229162 16484 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260926 04:01:34.229809 16502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:01:34.229996 16484 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 04:01:34.230095 16484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-1/wal
uuid: "c3b09a5fb725437ca6c07e2fedf6248d"
format_stamp: "Formatted at 2026-09-26 04:01:13 on dist-test-slave-nhpw"
server_key: "c288f9c4922358c61463c80840e6fc18"
server_key_iv: "6460d5e64d64769288126e4a2799ccb7"
server_key_version: "encryptionkey@0"
I20260926 04:01:34.230430 16484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:01:34.249313 16484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:01:34.251644 16484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:01:34.251813 16484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:01:34.252053 16484 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:01:34.252362 16484 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:01:34.252414 16484 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:01:34.252454 16484 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:01:34.252482 16484 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:01:34.258445 16484 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41269
I20260926 04:01:34.258493 16615 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41269 every 8 connection(s)
I20260926 04:01:34.258821 16484 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
Sep 26 04:01:34 dist-test-slave-nhpw krb5kdc[15709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395294, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 04:01:34.265341 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 16484
I20260926 04:01:34.265700 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:44153
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:32815
--webserver_port=44567
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 04:01:34.268289 16618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:34.259439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32815 (local address 127.0.98.2:57829)
0926 04:01:34.259671 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0926 04:01:34.259712 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 04:01:34.260230 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:01:34.260505 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:01:34.260509 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:01:34.261142 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:01:34.261328 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:01:34.261332 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:01:34.262037 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:01:34.262037 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:01:34.262798 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:01:34.262800 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:01:34.263182 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:01:34.263182 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:01:34.263283 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:01:34.263783 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:01:34.263792 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:01:34.265196 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:01:34.267045 (+ 1849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:01:34.267051 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:01:34.267053 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:01:34.267259 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:01:34.267478 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:01:34.267479 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:01:34.267479 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:01:34.267567 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:01:34.267837 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:01:34.267841 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:01:34.267946 (+ 105us) client_negotiation.cc:769] Sending connection context
0926 04:01:34.268026 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 04:01:34.268139 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":106,"threads_started":1}
I20260926 04:01:34.268286 16481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:34.259480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57829 (local address 127.0.98.62:32815)
0926 04:01:34.259598 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 04:01:34.259600 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:01:34.259836 (+ 236us) server_negotiation.cc:407] Connection header received
0926 04:01:34.260352 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:01:34.260353 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:01:34.260364 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:01:34.260431 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:01:34.261437 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:01:34.261950 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:01:34.262882 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:01:34.263094 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:01:34.263152 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:01:34.265288 (+ 2136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:01:34.265290 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:01:34.265291 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:01:34.265299 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:01:34.266952 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:01:34.267336 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:01:34.267338 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:01:34.267339 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:01:34.267411 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:01:34.267630 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:01:34.267631 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:01:34.267631 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:01:34.267774 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:01:34.267844 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 04:01:34.268070 (+ 226us) server_negotiation.cc:299] Negotiation successful
0926 04:01:34.268168 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 04:01:34.269047 16616 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32815
I20260926 04:01:34.269134 16616 heartbeater.cc:464] Registering TS with master...
I20260926 04:01:34.269310 16616 heartbeater.cc:511] Master 127.0.98.62:32815 requested a full tablet report, sending...
I20260926 04:01:34.269697 16290 ts_manager.cc:194] Registered new tserver with Master: c3b09a5fb725437ca6c07e2fedf6248d (127.0.98.2:41269)
I20260926 04:01:34.270254 16290 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57829
W20260926 04:01:34.346593 16619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:01:34.346793 16619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:01:34.346815 16619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:01:34.347855 16619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:01:34.347887 16619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:01:34.347913 16619 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 04:01:34.349257 16619 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:44153
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44567
--tserver_master_addrs=127.0.98.62:32815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16619
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:01:34.349428 16619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:01:34.349596 16619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 04:01:34.354828 16619 server_base.cc:1034] running on GCE node
W20260926 04:01:34.354784 16624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:01:34.354794 16625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:01:34.354782 16627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:01:34.355175 16619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:01:34.355365 16619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:01:34.356496 16619 hybrid_clock.cc:648] HybridClock initialized: now 1790395294356487 us; error 28 us; skew 500 ppm
Sep 26 04:01:34 dist-test-slave-nhpw krb5kdc[15709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395294, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:01:34.358903 16619 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 04:01:34.359270 16619 webserver.cc:460] Webserver started at http://127.0.98.3:44567/ using document root <none> and password file <none>
I20260926 04:01:34.359439 16619 fs_manager.cc:359] Metadata directory not provided
I20260926 04:01:34.359488 16619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:01:34.360782 16619 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 04:01:34.361389 16634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:01:34.361582 16619 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 04:01:34.361682 16619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-2/wal
uuid: "42019c7843344a1983c3fa631a898039"
format_stamp: "Formatted at 2026-09-26 04:01:13 on dist-test-slave-nhpw"
server_key: "11ce221d825db0c8fbd814c6e762cf7f"
server_key_iv: "9c6fbb99032088ef988deededac6b6eb"
server_key_version: "encryptionkey@0"
I20260926 04:01:34.361997 16619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:01:34.384543 16619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:01:34.386389 16619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:01:34.386545 16619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:01:34.386775 16619 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:01:34.387074 16619 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:01:34.387128 16619 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:01:34.387169 16619 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:01:34.387197 16619 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:01:34.392692 16619 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44153
I20260926 04:01:34.392728 16747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44153 every 8 connection(s)
I20260926 04:01:34.393079 16619 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
Sep 26 04:01:34 dist-test-slave-nhpw krb5kdc[15709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395294, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 04:01:34.401688 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 16619
I20260926 04:01:34.402738 16481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:34.393798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36781 (local address 127.0.98.62:32815)
0926 04:01:34.393888 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0926 04:01:34.393890 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:01:34.394044 (+ 154us) server_negotiation.cc:407] Connection header received
0926 04:01:34.394497 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:01:34.394498 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:01:34.394509 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:01:34.394573 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:01:34.395621 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:01:34.396111 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:01:34.397245 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:01:34.397431 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:01:34.397502 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:01:34.399552 (+ 2050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:01:34.399553 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:01:34.399554 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:01:34.399560 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:01:34.401079 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:01:34.401509 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:01:34.401510 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:01:34.401511 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:01:34.401567 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:01:34.401890 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:01:34.401892 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:01:34.401892 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:01:34.402071 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:01:34.402123 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 04:01:34.402545 (+ 422us) server_negotiation.cc:299] Negotiation successful
0926 04:01:34.402634 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 04:01:34.402777 16750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:34.393762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32815 (local address 127.0.98.3:36781)
0926 04:01:34.393920 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 04:01:34.393955 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 04:01:34.394414 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:01:34.394629 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:01:34.394632 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:01:34.395326 (+ 694us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:01:34.395531 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:01:34.395536 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:01:34.396214 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:01:34.396216 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:01:34.397153 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:01:34.397155 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:01:34.397550 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:01:34.397551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:01:34.397655 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:01:34.398115 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:01:34.398123 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:01:34.399461 (+ 1338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:01:34.401151 (+ 1690us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:01:34.401162 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:01:34.401164 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:01:34.401402 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:01:34.401635 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:01:34.401636 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:01:34.401637 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:01:34.401789 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:01:34.402178 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:01:34.402184 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:01:34.402370 (+ 186us) client_negotiation.cc:769] Sending connection context
0926 04:01:34.402487 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 04:01:34.402620 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":100,"threads_started":1}
I20260926 04:01:34.403335 16748 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32815
I20260926 04:01:34.403466 16748 heartbeater.cc:464] Registering TS with master...
I20260926 04:01:34.403681 16748 heartbeater.cc:511] Master 127.0.98.62:32815 requested a full tablet report, sending...
I20260926 04:01:34.404127 16290 ts_manager.cc:194] Registered new tserver with Master: 42019c7843344a1983c3fa631a898039 (127.0.98.3:44153)
I20260926 04:01:34.404506 16290 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36781
I20260926 04:01:34.409307 16752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:34.402101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50782 (local address 127.0.98.62:32815)
0926 04:01:34.402226 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 04:01:34.402228 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:01:34.402316 (+ 88us) server_negotiation.cc:407] Connection header received
0926 04:01:34.402345 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:01:34.402346 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:01:34.402358 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:01:34.402431 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:01:34.403557 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:01:34.404275 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:01:34.405177 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:01:34.405392 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:01:34.405477 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:01:34.406688 (+ 1211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:01:34.406689 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:01:34.406689 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:01:34.406696 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:01:34.408134 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:01:34.408486 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:01:34.408487 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:01:34.408488 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:01:34.408549 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:01:34.408782 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:01:34.408783 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:01:34.408783 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:01:34.408896 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:01:34.409079 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0926 04:01:34.409115 (+ 36us) server_negotiation.cc:299] Negotiation successful
0926 04:01:34.409195 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":77,"threads_started":1}
I20260926 04:01:34.410084 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 04:01:34.413061 16752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:34.410472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50792 (local address 127.0.98.62:32815)
0926 04:01:34.410505 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 04:01:34.410507 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:01:34.410811 (+ 304us) server_negotiation.cc:407] Connection header received
0926 04:01:34.410845 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:01:34.410846 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:01:34.410875 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:01:34.410942 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 04:01:34.411037 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:01:34.411523 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:01:34.412273 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:01:34.412469 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:01:34.412571 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:01:34.412827 (+ 256us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 04:01:34.412869 (+ 42us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0926 04:01:34.413012 (+ 143us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50792: token expired
Metrics: {"server-negotiator.queue_time_us":16}
W20260926 04:01:34.413144 16752 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50792: token expired
W20260926 04:01:34.413333 16217 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:32815)
I20260926 04:01:34.419559 16752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:34.413552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50794 (local address 127.0.98.62:32815)
0926 04:01:34.413608 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 04:01:34.413610 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:01:34.413662 (+ 52us) server_negotiation.cc:407] Connection header received
0926 04:01:34.413733 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:01:34.413734 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:01:34.413743 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:01:34.413798 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:01:34.414288 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:01:34.414757 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:01:34.415462 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:01:34.415655 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:01:34.415727 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:01:34.416952 (+ 1225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:01:34.416954 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:01:34.416954 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:01:34.416960 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:01:34.418424 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:01:34.418796 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:01:34.418797 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:01:34.418798 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:01:34.418859 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:01:34.419055 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:01:34.419056 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:01:34.419056 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:01:34.419174 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:01:34.419275 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 04:01:34.419388 (+ 113us) server_negotiation.cc:299] Negotiation successful
0926 04:01:34.419461 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 04:01:34.420758 16290 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50794:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 04:01:34.422276 16290 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 04:01:34.424508 16412 tablet_service.cc:1467] Processing CreateTablet for tablet bbb8316c021f4aafb27d2bbdc198ac5f (DEFAULT_TABLE table=test-table [id=8c114e5b03cf4eeaa574893242c8f5c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:01:34.424775 16412 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbb8316c021f4aafb27d2bbdc198ac5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:01:34.426095 16370 tablet_bootstrap.cc:492] T bbb8316c021f4aafb27d2bbdc198ac5f P ffccccc56a0949a29ec497d810c39c43: Bootstrap starting.
I20260926 04:01:34.426409 16370 tablet_bootstrap.cc:654] T bbb8316c021f4aafb27d2bbdc198ac5f P ffccccc56a0949a29ec497d810c39c43: Neither blocks nor log segments found. Creating new log.
I20260926 04:01:34.427309 16370 tablet_bootstrap.cc:492] T bbb8316c021f4aafb27d2bbdc198ac5f P ffccccc56a0949a29ec497d810c39c43: No bootstrap required, opened a new log
I20260926 04:01:34.427377 16370 ts_tablet_manager.cc:1397] T bbb8316c021f4aafb27d2bbdc198ac5f P ffccccc56a0949a29ec497d810c39c43: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 04:01:34.427541 16370 raft_consensus.cc:348] T bbb8316c021f4aafb27d2bbdc198ac5f P ffccccc56a0949a29ec497d810c39c43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42019c7843344a1983c3fa631a898039" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44153 } } peers { permanent_uuid: "c3b09a5fb725437ca6c07e2fedf6248d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41269 } } peers { permanent_uuid: "ffccccc56a0949a29ec497d810c39c43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39669 } }
I20260926 04:01:34.427621 16370 raft_consensus.cc:374] T bbb8316c021f4aafb27d2bbdc198ac5f P ffccccc56a0949a29ec497d810c39c43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:01:34.427675 16370 raft_consensus.cc:729] T bbb8316c021f4aafb27d2bbdc198ac5f P ffccccc56a0949a29ec497d810c39c43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffccccc56a0949a29ec497d810c39c43, State: Initialized, Role: FOLLOWER
I20260926 04:01:34.427718 16370 consensus_queue.cc:260] T bbb8316c021f4aafb27d2bbdc198ac5f P ffccccc56a0949a29ec497d810c39c43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42019c7843344a1983c3fa631a898039" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44153 } } peers { permanent_uuid: "c3b09a5fb725437ca6c07e2fedf6248d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41269 } } peers { permanent_uuid: "ffccccc56a0949a29ec497d810c39c43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39669 } }
I20260926 04:01:34.427814 16370 ts_tablet_manager.cc:1428] T bbb8316c021f4aafb27d2bbdc198ac5f P ffccccc56a0949a29ec497d810c39c43: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 04:01:34.428231 16485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:34.424129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44153 (local address 127.0.0.1:45534)
0926 04:01:34.424187 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 04:01:34.424198 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 04:01:34.424231 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:01:34.424660 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:01:34.424661 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:01:34.424663 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:01:34.424705 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:01:34.424707 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:01:34.425964 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:01:34.425965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:01:34.426987 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:01:34.426990 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:01:34.427944 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:01:34.427945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:01:34.428020 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:01:34.428043 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 04:01:34.428119 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 04:01:34.428165 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 04:01:34.428261 16755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:34.424220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45534 (local address 127.0.98.3:44153)
0926 04:01:34.424428 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 04:01:34.424433 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 04:01:34.424444 (+ 11us) server_negotiation.cc:407] Connection header received
0926 04:01:34.424496 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:01:34.424498 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:01:34.424517 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:01:34.424660 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:01:34.424759 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:01:34.425871 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:01:34.427080 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:01:34.427856 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:01:34.427924 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:01:34.427980 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 04:01:34.428140 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 04:01:34.428183 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":108,"threads_started":1}
I20260926 04:01:34.428555 16754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:34.424160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41269 (local address 127.0.0.1:40730)
0926 04:01:34.424544 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0926 04:01:34.424555 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 04:01:34.424605 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:01:34.424821 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:01:34.424822 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:01:34.424824 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:01:34.424855 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:01:34.424856 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:01:34.426129 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:01:34.426130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:01:34.427294 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:01:34.427297 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:01:34.428324 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:01:34.428324 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:01:34.428385 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:01:34.428389 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 04:01:34.428445 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 04:01:34.428486 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":109,"threads_started":1}
I20260926 04:01:34.428583 16756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:34.424353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40730 (local address 127.0.98.2:41269)
0926 04:01:34.424545 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 04:01:34.424550 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 04:01:34.424686 (+ 136us) server_negotiation.cc:407] Connection header received
0926 04:01:34.424730 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:01:34.424732 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:01:34.424749 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:01:34.424824 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:01:34.424907 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:01:34.426018 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:01:34.427464 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:01:34.428163 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:01:34.428229 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:01:34.428287 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 04:01:34.428459 (+ 172us) server_negotiation.cc:299] Negotiation successful
0926 04:01:34.428506 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":91,"threads_started":1}
I20260926 04:01:34.429019 16682 tablet_service.cc:1467] Processing CreateTablet for tablet bbb8316c021f4aafb27d2bbdc198ac5f (DEFAULT_TABLE table=test-table [id=8c114e5b03cf4eeaa574893242c8f5c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:01:34.429265 16550 tablet_service.cc:1467] Processing CreateTablet for tablet bbb8316c021f4aafb27d2bbdc198ac5f (DEFAULT_TABLE table=test-table [id=8c114e5b03cf4eeaa574893242c8f5c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:01:34.429388 16682 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbb8316c021f4aafb27d2bbdc198ac5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:01:34.429589 16550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbb8316c021f4aafb27d2bbdc198ac5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:01:34.432207 16758 tablet_bootstrap.cc:492] T bbb8316c021f4aafb27d2bbdc198ac5f P c3b09a5fb725437ca6c07e2fedf6248d: Bootstrap starting.
I20260926 04:01:34.432224 16759 tablet_bootstrap.cc:492] T bbb8316c021f4aafb27d2bbdc198ac5f P 42019c7843344a1983c3fa631a898039: Bootstrap starting.
I20260926 04:01:34.432861 16758 tablet_bootstrap.cc:654] T bbb8316c021f4aafb27d2bbdc198ac5f P c3b09a5fb725437ca6c07e2fedf6248d: Neither blocks nor log segments found. Creating new log.
I20260926 04:01:34.432973 16759 tablet_bootstrap.cc:654] T bbb8316c021f4aafb27d2bbdc198ac5f P 42019c7843344a1983c3fa631a898039: Neither blocks nor log segments found. Creating new log.
I20260926 04:01:34.433465 16759 log.cc:824] T bbb8316c021f4aafb27d2bbdc198ac5f P 42019c7843344a1983c3fa631a898039: Log is configured to *not* fsync() on all Append() calls
I20260926 04:01:34.433471 16758 log.cc:824] T bbb8316c021f4aafb27d2bbdc198ac5f P c3b09a5fb725437ca6c07e2fedf6248d: Log is configured to *not* fsync() on all Append() calls
I20260926 04:01:34.434384 16759 tablet_bootstrap.cc:492] T bbb8316c021f4aafb27d2bbdc198ac5f P 42019c7843344a1983c3fa631a898039: No bootstrap required, opened a new log
I20260926 04:01:34.434387 16758 tablet_bootstrap.cc:492] T bbb8316c021f4aafb27d2bbdc198ac5f P c3b09a5fb725437ca6c07e2fedf6248d: No bootstrap required, opened a new log
I20260926 04:01:34.434463 16758 ts_tablet_manager.cc:1397] T bbb8316c021f4aafb27d2bbdc198ac5f P c3b09a5fb725437ca6c07e2fedf6248d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 04:01:34.434464 16759 ts_tablet_manager.cc:1397] T bbb8316c021f4aafb27d2bbdc198ac5f P 42019c7843344a1983c3fa631a898039: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 04:01:34.435765 16758 raft_consensus.cc:348] T bbb8316c021f4aafb27d2bbdc198ac5f P c3b09a5fb725437ca6c07e2fedf6248d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42019c7843344a1983c3fa631a898039" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44153 } } peers { permanent_uuid: "c3b09a5fb725437ca6c07e2fedf6248d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41269 } } peers { permanent_uuid: "ffccccc56a0949a29ec497d810c39c43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39669 } }
I20260926 04:01:34.435765 16759 raft_consensus.cc:348] T bbb8316c021f4aafb27d2bbdc198ac5f P 42019c7843344a1983c3fa631a898039 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42019c7843344a1983c3fa631a898039" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44153 } } peers { permanent_uuid: "c3b09a5fb725437ca6c07e2fedf6248d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41269 } } peers { permanent_uuid: "ffccccc56a0949a29ec497d810c39c43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39669 } }
I20260926 04:01:34.435923 16759 raft_consensus.cc:374] T bbb8316c021f4aafb27d2bbdc198ac5f P 42019c7843344a1983c3fa631a898039 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:01:34.435927 16758 raft_consensus.cc:374] T bbb8316c021f4aafb27d2bbdc198ac5f P c3b09a5fb725437ca6c07e2fedf6248d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:01:34.435945 16759 raft_consensus.cc:729] T bbb8316c021f4aafb27d2bbdc198ac5f P 42019c7843344a1983c3fa631a898039 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42019c7843344a1983c3fa631a898039, State: Initialized, Role: FOLLOWER
I20260926 04:01:34.436002 16758 raft_consensus.cc:729] T bbb8316c021f4aafb27d2bbdc198ac5f P c3b09a5fb725437ca6c07e2fedf6248d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c3b09a5fb725437ca6c07e2fedf6248d, State: Initialized, Role: FOLLOWER
I20260926 04:01:34.436094 16759 consensus_queue.cc:260] T bbb8316c021f4aafb27d2bbdc198ac5f P 42019c7843344a1983c3fa631a898039 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42019c7843344a1983c3fa631a898039" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44153 } } peers { permanent_uuid: "c3b09a5fb725437ca6c07e2fedf6248d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41269 } } peers { permanent_uuid: "ffccccc56a0949a29ec497d810c39c43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39669 } }
I20260926 04:01:34.436122 16758 consensus_queue.cc:260] T bbb8316c021f4aafb27d2bbdc198ac5f P c3b09a5fb725437ca6c07e2fedf6248d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42019c7843344a1983c3fa631a898039" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44153 } } peers { permanent_uuid: "c3b09a5fb725437ca6c07e2fedf6248d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41269 } } peers { permanent_uuid: "ffccccc56a0949a29ec497d810c39c43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39669 } }
I20260926 04:01:34.436426 16616 heartbeater.cc:503] Master 127.0.98.62:32815 was elected leader, sending a full tablet report...
I20260926 04:01:34.436412 16748 heartbeater.cc:503] Master 127.0.98.62:32815 was elected leader, sending a full tablet report...
I20260926 04:01:34.436409 16759 ts_tablet_manager.cc:1428] T bbb8316c021f4aafb27d2bbdc198ac5f P 42019c7843344a1983c3fa631a898039: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 04:01:34.436697 16758 ts_tablet_manager.cc:1428] T bbb8316c021f4aafb27d2bbdc198ac5f P c3b09a5fb725437ca6c07e2fedf6248d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260926 04:01:34.509394 16617 tablet.cc:2369] T bbb8316c021f4aafb27d2bbdc198ac5f P c3b09a5fb725437ca6c07e2fedf6248d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 04:01:34.621615 16479 tablet.cc:2369] T bbb8316c021f4aafb27d2bbdc198ac5f P ffccccc56a0949a29ec497d810c39c43: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 04:01:34.631599 16763 raft_consensus.cc:482] T bbb8316c021f4aafb27d2bbdc198ac5f P c3b09a5fb725437ca6c07e2fedf6248d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 04:01:34.631814 16763 raft_consensus.cc:504] T bbb8316c021f4aafb27d2bbdc198ac5f P c3b09a5fb725437ca6c07e2fedf6248d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42019c7843344a1983c3fa631a898039" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44153 } } peers { permanent_uuid: "c3b09a5fb725437ca6c07e2fedf6248d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41269 } } peers { permanent_uuid: "ffccccc56a0949a29ec497d810c39c43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39669 } }
I20260926 04:01:34.632304 16763 leader_election.cc:288] T bbb8316c021f4aafb27d2bbdc198ac5f P c3b09a5fb725437ca6c07e2fedf6248d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 42019c7843344a1983c3fa631a898039 (127.0.98.3:44153), ffccccc56a0949a29ec497d810c39c43 (127.0.98.1:39669)
I20260926 04:01:34.635393 16755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:34.632668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47821 (local address 127.0.98.3:44153)
0926 04:01:34.632712 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 04:01:34.632714 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:01:34.632722 (+ 8us) server_negotiation.cc:407] Connection header received
0926 04:01:34.632752 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:01:34.632753 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:01:34.632780 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:01:34.632839 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:01:34.632997 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:01:34.633473 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:01:34.634574 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:01:34.635004 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:01:34.635108 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:01:34.635119 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 04:01:34.635284 (+ 165us) server_negotiation.cc:299] Negotiation successful
0926 04:01:34.635320 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 04:01:34.635418 16618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:34.632462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44153 (local address 127.0.98.2:47821)
0926 04:01:34.632549 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0926 04:01:34.632562 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 04:01:34.632609 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:01:34.632844 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:01:34.632845 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:01:34.632846 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:01:34.632916 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:01:34.632918 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:01:34.633558 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:01:34.633559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:01:34.634463 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:01:34.634465 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:01:34.635110 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:01:34.635111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:01:34.635193 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:01:34.635201 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 04:01:34.635276 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 04:01:34.635317 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 04:01:34.635430 16486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:34.632524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49341 (local address 127.0.98.1:39669)
0926 04:01:34.632589 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 04:01:34.632592 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:01:34.632745 (+ 153us) server_negotiation.cc:407] Connection header received
0926 04:01:34.632842 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:01:34.632843 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:01:34.632868 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:01:34.632966 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:01:34.633124 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:01:34.633617 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:01:34.634668 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:01:34.635076 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:01:34.635178 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:01:34.635254 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 04:01:34.635325 (+ 71us) server_negotiation.cc:299] Negotiation successful
0926 04:01:34.635359 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 04:01:34.635524 16764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:34.632476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39669 (local address 127.0.98.2:49341)
0926 04:01:34.632685 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0926 04:01:34.632696 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 04:01:34.632765 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:01:34.632972 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:01:34.632973 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:01:34.632974 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:01:34.633044 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:01:34.633046 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:01:34.633732 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:01:34.633733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:01:34.634563 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:01:34.634565 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:01:34.635186 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:01:34.635187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:01:34.635253 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:01:34.635259 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 04:01:34.635319 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 04:01:34.635358 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":102,"threads_started":1}
I20260926 04:01:34.635761 16702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbb8316c021f4aafb27d2bbdc198ac5f" candidate_uuid: "c3b09a5fb725437ca6c07e2fedf6248d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42019c7843344a1983c3fa631a898039" is_pre_election: true
I20260926 04:01:34.635914 16702 raft_consensus.cc:2454] T bbb8316c021f4aafb27d2bbdc198ac5f P 42019c7843344a1983c3fa631a898039 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3b09a5fb725437ca6c07e2fedf6248d in term 0.
I20260926 04:01:34.636022 16432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbb8316c021f4aafb27d2bbdc198ac5f" candidate_uuid: "c3b09a5fb725437ca6c07e2fedf6248d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffccccc56a0949a29ec497d810c39c43" is_pre_election: true
I20260926 04:01:34.636170 16432 raft_consensus.cc:2454] T bbb8316c021f4aafb27d2bbdc198ac5f P ffccccc56a0949a29ec497d810c39c43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c3b09a5fb725437ca6c07e2fedf6248d in term 0.
I20260926 04:01:34.636133 16506 leader_election.cc:302] T bbb8316c021f4aafb27d2bbdc198ac5f P c3b09a5fb725437ca6c07e2fedf6248d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42019c7843344a1983c3fa631a898039, c3b09a5fb725437ca6c07e2fedf6248d; no voters:
I20260926 04:01:34.636283 16763 raft_consensus.cc:2781] T bbb8316c021f4aafb27d2bbdc198ac5f P c3b09a5fb725437ca6c07e2fedf6248d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 04:01:34.636344 16763 raft_consensus.cc:482] T bbb8316c021f4aafb27d2bbdc198ac5f P c3b09a5fb725437ca6c07e2fedf6248d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 04:01:34.636373 16763 raft_consensus.cc:3037] T bbb8316c021f4aafb27d2bbdc198ac5f P c3b09a5fb725437ca6c07e2fedf6248d [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:01:34.637138 16763 raft_consensus.cc:504] T bbb8316c021f4aafb27d2bbdc198ac5f P c3b09a5fb725437ca6c07e2fedf6248d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42019c7843344a1983c3fa631a898039" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44153 } } peers { permanent_uuid: "c3b09a5fb725437ca6c07e2fedf6248d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41269 } } peers { permanent_uuid: "ffccccc56a0949a29ec497d810c39c43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39669 } }
I20260926 04:01:34.637277 16763 leader_election.cc:288] T bbb8316c021f4aafb27d2bbdc198ac5f P c3b09a5fb725437ca6c07e2fedf6248d [CANDIDATE]: Term 1 election: Requested vote from peers 42019c7843344a1983c3fa631a898039 (127.0.98.3:44153), ffccccc56a0949a29ec497d810c39c43 (127.0.98.1:39669)
I20260926 04:01:34.637428 16702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbb8316c021f4aafb27d2bbdc198ac5f" candidate_uuid: "c3b09a5fb725437ca6c07e2fedf6248d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42019c7843344a1983c3fa631a898039"
I20260926 04:01:34.637445 16432 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbb8316c021f4aafb27d2bbdc198ac5f" candidate_uuid: "c3b09a5fb725437ca6c07e2fedf6248d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ffccccc56a0949a29ec497d810c39c43"
I20260926 04:01:34.637513 16702 raft_consensus.cc:3037] T bbb8316c021f4aafb27d2bbdc198ac5f P 42019c7843344a1983c3fa631a898039 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:01:34.637531 16432 raft_consensus.cc:3037] T bbb8316c021f4aafb27d2bbdc198ac5f P ffccccc56a0949a29ec497d810c39c43 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:01:34.638118 16702 raft_consensus.cc:2454] T bbb8316c021f4aafb27d2bbdc198ac5f P 42019c7843344a1983c3fa631a898039 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3b09a5fb725437ca6c07e2fedf6248d in term 1.
I20260926 04:01:34.638234 16432 raft_consensus.cc:2454] T bbb8316c021f4aafb27d2bbdc198ac5f P ffccccc56a0949a29ec497d810c39c43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c3b09a5fb725437ca6c07e2fedf6248d in term 1.
I20260926 04:01:34.638301 16506 leader_election.cc:302] T bbb8316c021f4aafb27d2bbdc198ac5f P c3b09a5fb725437ca6c07e2fedf6248d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42019c7843344a1983c3fa631a898039, c3b09a5fb725437ca6c07e2fedf6248d; no voters:
I20260926 04:01:34.638381 16763 raft_consensus.cc:2781] T bbb8316c021f4aafb27d2bbdc198ac5f P c3b09a5fb725437ca6c07e2fedf6248d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:01:34.638505 16763 raft_consensus.cc:686] T bbb8316c021f4aafb27d2bbdc198ac5f P c3b09a5fb725437ca6c07e2fedf6248d [term 1 LEADER]: Becoming Leader. State: Replica: c3b09a5fb725437ca6c07e2fedf6248d, State: Running, Role: LEADER
I20260926 04:01:34.638599 16763 consensus_queue.cc:237] T bbb8316c021f4aafb27d2bbdc198ac5f P c3b09a5fb725437ca6c07e2fedf6248d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42019c7843344a1983c3fa631a898039" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44153 } } peers { permanent_uuid: "c3b09a5fb725437ca6c07e2fedf6248d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41269 } } peers { permanent_uuid: "ffccccc56a0949a29ec497d810c39c43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39669 } }
I20260926 04:01:34.639406 16290 catalog_manager.cc:5495] T bbb8316c021f4aafb27d2bbdc198ac5f P c3b09a5fb725437ca6c07e2fedf6248d reported cstate change: term changed from 0 to 1, leader changed from <none> to c3b09a5fb725437ca6c07e2fedf6248d (127.0.98.2). New cstate: current_term: 1 leader_uuid: "c3b09a5fb725437ca6c07e2fedf6248d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42019c7843344a1983c3fa631a898039" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c3b09a5fb725437ca6c07e2fedf6248d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41269 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ffccccc56a0949a29ec497d810c39c43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39669 } health_report { overall_health: UNKNOWN } } }
W20260926 04:01:34.643726 16749 tablet.cc:2369] T bbb8316c021f4aafb27d2bbdc198ac5f P 42019c7843344a1983c3fa631a898039: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 04:01:34.657021 16756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:34.654687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40732 (local address 127.0.98.2:41269)
0926 04:01:34.654752 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 04:01:34.654754 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:01:34.654762 (+ 8us) server_negotiation.cc:407] Connection header received
0926 04:01:34.654796 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:01:34.654796 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:01:34.654813 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:01:34.654874 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 04:01:34.654988 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:01:34.655450 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:01:34.656193 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:01:34.656377 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:01:34.656438 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:01:34.656609 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 04:01:34.656772 (+ 163us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 04:01:34.656848 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 04:01:34.656886 (+ 38us) server_negotiation.cc:299] Negotiation successful
0926 04:01:34.656904 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 04:01:34.658843 16702 raft_consensus.cc:1261] T bbb8316c021f4aafb27d2bbdc198ac5f P 42019c7843344a1983c3fa631a898039 [term 1 FOLLOWER]: Refusing update from remote peer c3b09a5fb725437ca6c07e2fedf6248d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:01:34.658855 16432 raft_consensus.cc:1261] T bbb8316c021f4aafb27d2bbdc198ac5f P ffccccc56a0949a29ec497d810c39c43 [term 1 FOLLOWER]: Refusing update from remote peer c3b09a5fb725437ca6c07e2fedf6248d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:01:34.659202 16763 consensus_queue.cc:1035] T bbb8316c021f4aafb27d2bbdc198ac5f P c3b09a5fb725437ca6c07e2fedf6248d [LEADER]: Connected to new peer: Peer: permanent_uuid: "42019c7843344a1983c3fa631a898039" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:01:34.659354 16765 consensus_queue.cc:1035] T bbb8316c021f4aafb27d2bbdc198ac5f P c3b09a5fb725437ca6c07e2fedf6248d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ffccccc56a0949a29ec497d810c39c43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39669 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:01:34.661419 16769 mvcc.cc:205] Tried to move back new op lower bound from 7333459126919811072 to 7333459126840250368. Current Snapshot: MvccSnapshot[applied={T|T < 7333459126919811072}]
I20260926 04:01:34.661991 16770 mvcc.cc:205] Tried to move back new op lower bound from 7333459126919811072 to 7333459126840250368. Current Snapshot: MvccSnapshot[applied={T|T < 7333459126919811072}]
I20260926 04:01:34.663595 16290 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50794:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 04:01:34.663709 16290 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50794:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 04:01:34.664536 16290 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5ad1f0dd809a452ca0355fe421f1b3ad: Sending DeleteTablet for 3 replicas of tablet bbb8316c021f4aafb27d2bbdc198ac5f
I20260926 04:01:34.664877 16550 tablet_service.cc:1514] Processing DeleteTablet for tablet bbb8316c021f4aafb27d2bbdc198ac5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:01:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40730
I20260926 04:01:34.664920 16412 tablet_service.cc:1514] Processing DeleteTablet for tablet bbb8316c021f4aafb27d2bbdc198ac5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:01:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38276
I20260926 04:01:34.665149 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 16328
I20260926 04:01:34.665197 16682 tablet_service.cc:1514] Processing DeleteTablet for tablet bbb8316c021f4aafb27d2bbdc198ac5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:01:34 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45534
I20260926 04:01:34.665807 16779 tablet_replica.cc:330] T bbb8316c021f4aafb27d2bbdc198ac5f P 42019c7843344a1983c3fa631a898039: stopping tablet replica
I20260926 04:01:34.665901 16779 raft_consensus.cc:2229] T bbb8316c021f4aafb27d2bbdc198ac5f P 42019c7843344a1983c3fa631a898039 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 04:01:34.665999 16779 raft_consensus.cc:2258] T bbb8316c021f4aafb27d2bbdc198ac5f P 42019c7843344a1983c3fa631a898039 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:01:34.666224 16777 tablet_replica.cc:330] T bbb8316c021f4aafb27d2bbdc198ac5f P c3b09a5fb725437ca6c07e2fedf6248d: stopping tablet replica
I20260926 04:01:34.666307 16777 raft_consensus.cc:2229] T bbb8316c021f4aafb27d2bbdc198ac5f P c3b09a5fb725437ca6c07e2fedf6248d [term 1 LEADER]: Raft consensus shutting down.
I20260926 04:01:34.666393 16777 raft_consensus.cc:2258] T bbb8316c021f4aafb27d2bbdc198ac5f P c3b09a5fb725437ca6c07e2fedf6248d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:01:34.666844 16777 ts_tablet_manager.cc:1905] T bbb8316c021f4aafb27d2bbdc198ac5f P c3b09a5fb725437ca6c07e2fedf6248d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:01:34.667057 16779 ts_tablet_manager.cc:1905] T bbb8316c021f4aafb27d2bbdc198ac5f P 42019c7843344a1983c3fa631a898039: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:01:34.669082 16777 ts_tablet_manager.cc:1918] T bbb8316c021f4aafb27d2bbdc198ac5f P c3b09a5fb725437ca6c07e2fedf6248d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 04:01:34.669091 16779 ts_tablet_manager.cc:1918] T bbb8316c021f4aafb27d2bbdc198ac5f P 42019c7843344a1983c3fa631a898039: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 04:01:34.669163 16777 log.cc:1196] T bbb8316c021f4aafb27d2bbdc198ac5f P c3b09a5fb725437ca6c07e2fedf6248d: Deleting WAL directory at /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-1/wal/wals/bbb8316c021f4aafb27d2bbdc198ac5f
I20260926 04:01:34.669294 16779 log.cc:1196] T bbb8316c021f4aafb27d2bbdc198ac5f P 42019c7843344a1983c3fa631a898039: Deleting WAL directory at /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-2/wal/wals/bbb8316c021f4aafb27d2bbdc198ac5f
I20260926 04:01:34.669450 16777 ts_tablet_manager.cc:1939] T bbb8316c021f4aafb27d2bbdc198ac5f P c3b09a5fb725437ca6c07e2fedf6248d: Deleting consensus metadata
I20260926 04:01:34.669710 16779 ts_tablet_manager.cc:1939] T bbb8316c021f4aafb27d2bbdc198ac5f P 42019c7843344a1983c3fa631a898039: Deleting consensus metadata
I20260926 04:01:34.669999 16276 catalog_manager.cc:4841] TS c3b09a5fb725437ca6c07e2fedf6248d (127.0.98.2:41269): tablet bbb8316c021f4aafb27d2bbdc198ac5f (table test-table [id=8c114e5b03cf4eeaa574893242c8f5c3]) successfully deleted
I20260926 04:01:34.670189 16277 catalog_manager.cc:4841] TS 42019c7843344a1983c3fa631a898039 (127.0.98.3:44153): tablet bbb8316c021f4aafb27d2bbdc198ac5f (table test-table [id=8c114e5b03cf4eeaa574893242c8f5c3]) successfully deleted
W20260926 04:01:34.671492 16275 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39669 (error 108)
I20260926 04:01:34.671773 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 16484
I20260926 04:01:34.672535 16754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:34.671909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39669 (local address 127.0.0.1:38286)
0926 04:01:34.672455 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0926 04:01:34.672496 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39669: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":522}
W20260926 04:01:34.672693 16275 catalog_manager.cc:4584] TS ffccccc56a0949a29ec497d810c39c43 (127.0.98.1:39669): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bbb8316c021f4aafb27d2bbdc198ac5f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39669: connect: Connection refused (error 111)
I20260926 04:01:34.677186 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 16619
I20260926 04:01:34.682126 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 16257
I20260926 04:01:54.689429 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32815
--webserver_interface=127.0.98.62
--webserver_port=40033
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32815
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:01:54.771356 16781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:01:54.771543 16781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:01:54.771565 16781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:01:54.772876 16781 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 04:01:54.772954 16781 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 04:01:54.772979 16781 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 04:01:54.773007 16781 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:01:54.773027 16781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:01:54.773056 16781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:01:54.773069 16781 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:01:54.773082 16781 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:01:54.774576 16781 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32815
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=40033
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16781
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:01:54.774760 16781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:01:54.774917 16781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:01:54.780311 16786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:01:54.780404 16789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:01:54.780373 16781 server_base.cc:1034] running on GCE node
W20260926 04:01:54.780385 16787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:01:54.780821 16781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:01:54.781054 16781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:01:54.782212 16781 hybrid_clock.cc:648] HybridClock initialized: now 1790395314782198 us; error 48 us; skew 500 ppm
Sep 26 04:01:54 dist-test-slave-nhpw krb5kdc[15709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395314, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:01:54.791231 16781 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 04:01:54.791745 16781 webserver.cc:460] Webserver started at http://127.0.98.62:40033/ using document root <none> and password file <none>
I20260926 04:01:54.791997 16781 fs_manager.cc:359] Metadata directory not provided
I20260926 04:01:54.792070 16781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:01:54.793573 16781 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 04:01:54.794346 16796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:01:54.794548 16781 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 04:01:54.794659 16781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/master-0/wal
uuid: "5ad1f0dd809a452ca0355fe421f1b3ad"
format_stamp: "Formatted at 2026-09-26 04:01:13 on dist-test-slave-nhpw"
server_key: "e81730a0b3966cb0eb8ead6f01601210"
server_key_iv: "08b0aaa8da8b10f6beb5f6eca5a53cf2"
server_key_version: "encryptionkey@0"
I20260926 04:01:54.795027 16781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:01:54.824961 16781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:01:54.827001 16781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:01:54.827189 16781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:01:54.831336 16781 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32815
I20260926 04:01:54.831370 16848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32815 every 8 connection(s)
I20260926 04:01:54.831743 16781 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
I20260926 04:01:54.833294 16849 sys_catalog.cc:263] Verifying existing consensus state
I20260926 04:01:54.833861 16849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ad1f0dd809a452ca0355fe421f1b3ad: Bootstrap starting.
I20260926 04:01:54.834520 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 16781
I20260926 04:01:54.834791 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:39669
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:32815
--webserver_port=38119
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 04:01:54.842638 16849 log.cc:824] T 00000000000000000000000000000000 P 5ad1f0dd809a452ca0355fe421f1b3ad: Log is configured to *not* fsync() on all Append() calls
I20260926 04:01:54.845005 16849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ad1f0dd809a452ca0355fe421f1b3ad: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 04:01:54.845342 16849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ad1f0dd809a452ca0355fe421f1b3ad: Bootstrap complete.
I20260926 04:01:54.847518 16849 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ad1f0dd809a452ca0355fe421f1b3ad [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ad1f0dd809a452ca0355fe421f1b3ad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32815 } }
I20260926 04:01:54.847734 16849 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ad1f0dd809a452ca0355fe421f1b3ad [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ad1f0dd809a452ca0355fe421f1b3ad, State: Initialized, Role: FOLLOWER
I20260926 04:01:54.847901 16849 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ad1f0dd809a452ca0355fe421f1b3ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ad1f0dd809a452ca0355fe421f1b3ad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32815 } }
I20260926 04:01:54.847975 16849 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ad1f0dd809a452ca0355fe421f1b3ad [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:01:54.848038 16849 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ad1f0dd809a452ca0355fe421f1b3ad [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:01:54.848083 16849 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ad1f0dd809a452ca0355fe421f1b3ad [term 2 FOLLOWER]: Advancing to term 3
I20260926 04:01:54.849155 16849 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ad1f0dd809a452ca0355fe421f1b3ad [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ad1f0dd809a452ca0355fe421f1b3ad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32815 } }
I20260926 04:01:54.849246 16849 leader_election.cc:302] T 00000000000000000000000000000000 P 5ad1f0dd809a452ca0355fe421f1b3ad [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ad1f0dd809a452ca0355fe421f1b3ad; no voters:
I20260926 04:01:54.849479 16849 leader_election.cc:288] T 00000000000000000000000000000000 P 5ad1f0dd809a452ca0355fe421f1b3ad [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 04:01:54.849546 16853 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ad1f0dd809a452ca0355fe421f1b3ad [term 3 FOLLOWER]: Leader election won for term 3
I20260926 04:01:54.849704 16853 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ad1f0dd809a452ca0355fe421f1b3ad [term 3 LEADER]: Becoming Leader. State: Replica: 5ad1f0dd809a452ca0355fe421f1b3ad, State: Running, Role: LEADER
I20260926 04:01:54.849742 16849 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ad1f0dd809a452ca0355fe421f1b3ad [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:01:54.849805 16853 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ad1f0dd809a452ca0355fe421f1b3ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ad1f0dd809a452ca0355fe421f1b3ad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32815 } }
I20260926 04:01:54.850063 16853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ad1f0dd809a452ca0355fe421f1b3ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ad1f0dd809a452ca0355fe421f1b3ad. Latest consensus state: current_term: 3 leader_uuid: "5ad1f0dd809a452ca0355fe421f1b3ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ad1f0dd809a452ca0355fe421f1b3ad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32815 } } }
I20260926 04:01:54.850065 16854 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ad1f0dd809a452ca0355fe421f1b3ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "5ad1f0dd809a452ca0355fe421f1b3ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ad1f0dd809a452ca0355fe421f1b3ad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32815 } } }
I20260926 04:01:54.850153 16853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ad1f0dd809a452ca0355fe421f1b3ad [sys.catalog]: This master's current role is: LEADER
I20260926 04:01:54.850178 16854 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ad1f0dd809a452ca0355fe421f1b3ad [sys.catalog]: This master's current role is: LEADER
I20260926 04:01:54.850610 16859 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:01:54.851359 16859 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:01:54.851456 16859 catalog_manager.cc:1259] Loaded cluster ID: f52174d07b3f4bb79225777d91d151b2
I20260926 04:01:54.851492 16859 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 04:01:54.852744 16859 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:01:54.852917 16859 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5ad1f0dd809a452ca0355fe421f1b3ad: Loaded TSK: 0
I20260926 04:01:54.852962 16859 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5ad1f0dd809a452ca0355fe421f1b3ad: Loaded TSK: 1
F20260926 04:01:54.853180 16859 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f85cf6259ad google::LogMessage::Fail() at ??:0
@ 0x7f85cf62888a google::LogMessage::SendToLog() at ??:0
@ 0x7f85cf6254de google::LogMessage::Flush() at ??:0
@ 0x7f85cf626a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f85d1d92104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f85d1d8fd41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f85d75831cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f85d758348e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f85d7583594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f85d7549c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f85d7553ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f85d14f30d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f85d14e8954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f85d234c6db start_thread at ??:0
@ 0x7f85ce12361f clone at ??:0
W20260926 04:01:54.936425 16851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:01:54.936694 16851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:01:54.936738 16851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:01:54.937949 16851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:01:54.938022 16851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:01:54.938068 16851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 04:01:54.939539 16851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:39669
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=38119
--tserver_master_addrs=127.0.98.62:32815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:01:54.939819 16851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:01:54.940059 16851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:01:54.945478 16890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:01:54.945546 16891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:01:54.945650 16851 server_base.cc:1034] running on GCE node
W20260926 04:01:54.945478 16893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:01:54.945966 16851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:01:54.946245 16851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:01:54.947407 16851 hybrid_clock.cc:648] HybridClock initialized: now 1790395314947398 us; error 30 us; skew 500 ppm
Sep 26 04:01:54 dist-test-slave-nhpw krb5kdc[15709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395314, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:01:54.949952 16851 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 04:01:54.950439 16851 webserver.cc:460] Webserver started at http://127.0.98.1:38119/ using document root <none> and password file <none>
I20260926 04:01:54.950685 16851 fs_manager.cc:359] Metadata directory not provided
I20260926 04:01:54.950734 16851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:01:54.952091 16851 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:01:54.953322 16900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:01:54.953504 16851 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 04:01:54.953619 16851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-0/wal
uuid: "ffccccc56a0949a29ec497d810c39c43"
format_stamp: "Formatted at 2026-09-26 04:01:13 on dist-test-slave-nhpw"
server_key: "88374dfc5922c3417e129b86cdc294a1"
server_key_iv: "b4eb3d1e4dc1023e48409622dae899da"
server_key_version: "encryptionkey@0"
I20260926 04:01:54.954000 16851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:01:54.966679 16851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:01:54.969110 16851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:01:54.969264 16851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:01:54.969563 16851 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:01:54.970094 16907 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 04:01:54.971536 16851 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 04:01:54.971590 16851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 04:01:54.971616 16851 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 04:01:54.972297 16851 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 04:01:54.972338 16851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 04:01:54.972391 16907 tablet_bootstrap.cc:492] T bbb8316c021f4aafb27d2bbdc198ac5f P ffccccc56a0949a29ec497d810c39c43: Bootstrap starting.
I20260926 04:01:54.980125 16851 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39669
I20260926 04:01:54.980345 17014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39669 every 8 connection(s)
I20260926 04:01:54.980672 16851 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
I20260926 04:01:54.983636 16907 log.cc:824] T bbb8316c021f4aafb27d2bbdc198ac5f P ffccccc56a0949a29ec497d810c39c43: Log is configured to *not* fsync() on all Append() calls
I20260926 04:01:54.983954 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 16851
I20260926 04:01:54.984283 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:41269
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:32815
--webserver_port=46101
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 04:01:54.986356 16907 tablet_bootstrap.cc:492] T bbb8316c021f4aafb27d2bbdc198ac5f P ffccccc56a0949a29ec497d810c39c43: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 04:01:54.986647 16907 tablet_bootstrap.cc:492] T bbb8316c021f4aafb27d2bbdc198ac5f P ffccccc56a0949a29ec497d810c39c43: Bootstrap complete.
I20260926 04:01:54.987572 16907 ts_tablet_manager.cc:1397] T bbb8316c021f4aafb27d2bbdc198ac5f P ffccccc56a0949a29ec497d810c39c43: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.009s
I20260926 04:01:54.988574 16907 raft_consensus.cc:348] T bbb8316c021f4aafb27d2bbdc198ac5f P ffccccc56a0949a29ec497d810c39c43 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42019c7843344a1983c3fa631a898039" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44153 } } peers { permanent_uuid: "c3b09a5fb725437ca6c07e2fedf6248d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41269 } } peers { permanent_uuid: "ffccccc56a0949a29ec497d810c39c43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39669 } }
I20260926 04:01:54.988912 16907 raft_consensus.cc:729] T bbb8316c021f4aafb27d2bbdc198ac5f P ffccccc56a0949a29ec497d810c39c43 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffccccc56a0949a29ec497d810c39c43, State: Initialized, Role: FOLLOWER
I20260926 04:01:54.989105 16907 consensus_queue.cc:260] T bbb8316c021f4aafb27d2bbdc198ac5f P ffccccc56a0949a29ec497d810c39c43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42019c7843344a1983c3fa631a898039" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44153 } } peers { permanent_uuid: "c3b09a5fb725437ca6c07e2fedf6248d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41269 } } peers { permanent_uuid: "ffccccc56a0949a29ec497d810c39c43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39669 } }
I20260926 04:01:54.989403 16907 ts_tablet_manager.cc:1428] T bbb8316c021f4aafb27d2bbdc198ac5f P ffccccc56a0949a29ec497d810c39c43: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260926 04:01:55.075409 17017 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:01:54.981432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32815 (local address 127.0.98.1:53393)
0926 04:01:54.981663 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0926 04:01:54.981713 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 04:01:54.982567 (+ 854us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:01:55.075181 (+ 92614us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32815: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":111,"threads_started":1}
W20260926 04:01:55.075662 17015 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:32815 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:32815: Client connection negotiation failed: client connection to 127.0.98.62:32815: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 04:01:55.076043 17017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:55.075932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32815 (local address 127.0.98.1:50299)
0926 04:01:55.075986 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 04:01:55.076026 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 04:01:55.076361 17017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:55.076280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32815 (local address 127.0.98.1:38633)
0926 04:01:55.076325 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 04:01:55.076347 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 04:01:55.076471 17015 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 04:01:55.088297 17019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:01:55.088544 17019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:01:55.088583 17019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:01:55.089773 17019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:01:55.089847 17019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:01:55.089893 17019 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 04:01:55.091432 17019 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:41269
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=46101
--tserver_master_addrs=127.0.98.62:32815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17019
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:01:55.091693 17019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:01:55.091933 17019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:01:55.097278 17025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:01:55.097344 17026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:01:55.097324 17019 server_base.cc:1034] running on GCE node
W20260926 04:01:55.097282 17028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:01:55.097641 17019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:01:55.097915 17019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:01:55.099103 17019 hybrid_clock.cc:648] HybridClock initialized: now 1790395315099079 us; error 41 us; skew 500 ppm
Sep 26 04:01:55 dist-test-slave-nhpw krb5kdc[15709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395315, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:01:55.101972 17019 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 04:01:55.102425 17019 webserver.cc:460] Webserver started at http://127.0.98.2:46101/ using document root <none> and password file <none>
I20260926 04:01:55.102654 17019 fs_manager.cc:359] Metadata directory not provided
I20260926 04:01:55.102725 17019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:01:55.104058 17019 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:01:55.104763 17035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:01:55.104943 17019 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:01:55.105039 17019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-1/wal
uuid: "c3b09a5fb725437ca6c07e2fedf6248d"
format_stamp: "Formatted at 2026-09-26 04:01:13 on dist-test-slave-nhpw"
server_key: "c288f9c4922358c61463c80840e6fc18"
server_key_iv: "6460d5e64d64769288126e4a2799ccb7"
server_key_version: "encryptionkey@0"
I20260926 04:01:55.105456 17019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:01:55.126372 17019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:01:55.128495 17019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:01:55.128757 17019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:01:55.129027 17019 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:01:55.129338 17019 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:01:55.129396 17019 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:01:55.129462 17019 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:01:55.129493 17019 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:01:55.135392 17019 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41269
I20260926 04:01:55.135478 17148 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41269 every 8 connection(s)
I20260926 04:01:55.135816 17019 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
I20260926 04:01:55.136905 17151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:55.136506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32815 (local address 127.0.98.2:60677)
0926 04:01:55.136676 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 04:01:55.136743 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":75,"threads_started":1}
W20260926 04:01:55.137290 17149 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:32815 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:32815: Client connection negotiation failed: client connection to 127.0.98.62:32815: connect: Connection refused (error 111)
I20260926 04:01:55.137573 17151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:55.137485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32815 (local address 127.0.98.2:46543)
0926 04:01:55.137532 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 04:01:55.137556 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 04:01:55.137837 17151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:55.137785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32815 (local address 127.0.98.2:43715)
0926 04:01:55.137807 (+ 22us) negotiation.cc:107] Waiting for socket to connect
0926 04:01:55.137825 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260926 04:01:55.137966 17149 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 04:01:55.141853 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 17019
I20260926 04:01:55.142072 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:44153
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:32815
--webserver_port=44567
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:01:55.221616 17152 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:01:55.221876 17152 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:01:55.221915 17152 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:01:55.223062 17152 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:01:55.223137 17152 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:01:55.223182 17152 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 04:01:55.224658 17152 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:44153
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44567
--tserver_master_addrs=127.0.98.62:32815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17152
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:01:55.224936 17152 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:01:55.225162 17152 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:01:55.230264 17158 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:01:55.230316 17157 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:01:55.230340 17152 server_base.cc:1034] running on GCE node
W20260926 04:01:55.230228 17160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:01:55.230633 17152 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:01:55.230875 17152 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:01:55.232025 17152 hybrid_clock.cc:648] HybridClock initialized: now 1790395315232017 us; error 34 us; skew 500 ppm
Sep 26 04:01:55 dist-test-slave-nhpw krb5kdc[15709](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395315, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:01:55.234586 17152 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 04:01:55.235034 17152 webserver.cc:460] Webserver started at http://127.0.98.3:44567/ using document root <none> and password file <none>
I20260926 04:01:55.235268 17152 fs_manager.cc:359] Metadata directory not provided
I20260926 04:01:55.235340 17152 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:01:55.236816 17152 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:01:55.237426 17167 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:01:55.237597 17152 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 04:01:55.237712 17152 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-2/wal
uuid: "42019c7843344a1983c3fa631a898039"
format_stamp: "Formatted at 2026-09-26 04:01:13 on dist-test-slave-nhpw"
server_key: "11ce221d825db0c8fbd814c6e762cf7f"
server_key_iv: "9c6fbb99032088ef988deededac6b6eb"
server_key_version: "encryptionkey@0"
I20260926 04:01:55.238067 17152 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:01:55.267002 17152 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:01:55.269096 17152 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:01:55.269261 17152 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:01:55.269496 17152 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:01:55.269819 17152 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:01:55.269876 17152 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:01:55.269918 17152 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:01:55.269946 17152 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:01:55.276335 17152 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44153
I20260926 04:01:55.276391 17280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44153 every 8 connection(s)
I20260926 04:01:55.276853 17152 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
I20260926 04:01:55.277958 17283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:55.277576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32815 (local address 127.0.98.3:44641)
0926 04:01:55.277752 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 04:01:55.277824 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":64,"threads_started":1}
W20260926 04:01:55.278390 17281 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:32815 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:32815: Client connection negotiation failed: client connection to 127.0.98.62:32815: connect: Connection refused (error 111)
I20260926 04:01:55.278733 17283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:55.278611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32815 (local address 127.0.98.3:53233)
0926 04:01:55.278682 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0926 04:01:55.278712 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260926 04:01:55.279044 17283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:55.278961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32815 (local address 127.0.98.3:57803)
0926 04:01:55.279005 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 04:01:55.279028 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 04:01:55.279176 17281 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 04:01:55.286949 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 17152
I20260926 04:01:56.077112 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:56.076884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32815 (local address 127.0.98.1:50353)
0926 04:01:56.077014 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 04:01:56.077051 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":58,"threads_started":1}
I20260926 04:01:56.138514 17286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:56.138312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32815 (local address 127.0.98.2:42157)
0926 04:01:56.138428 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0926 04:01:56.138462 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":59,"threads_started":1}
I20260926 04:01:56.279712 17287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:56.279507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32815 (local address 127.0.98.3:47455)
0926 04:01:56.279632 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 04:01:56.279670 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":55,"threads_started":1}
I20260926 04:01:56.558414 17288 raft_consensus.cc:482] T bbb8316c021f4aafb27d2bbdc198ac5f P ffccccc56a0949a29ec497d810c39c43 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 04:01:56.558532 17288 raft_consensus.cc:504] T bbb8316c021f4aafb27d2bbdc198ac5f P ffccccc56a0949a29ec497d810c39c43 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42019c7843344a1983c3fa631a898039" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44153 } } peers { permanent_uuid: "c3b09a5fb725437ca6c07e2fedf6248d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41269 } } peers { permanent_uuid: "ffccccc56a0949a29ec497d810c39c43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39669 } }
I20260926 04:01:56.558881 17288 leader_election.cc:288] T bbb8316c021f4aafb27d2bbdc198ac5f P ffccccc56a0949a29ec497d810c39c43 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42019c7843344a1983c3fa631a898039 (127.0.98.3:44153), c3b09a5fb725437ca6c07e2fedf6248d (127.0.98.2:41269)
Sep 26 04:01:56 dist-test-slave-nhpw krb5kdc[15709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395314, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 26 04:01:56 dist-test-slave-nhpw krb5kdc[15709](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395314, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260926 04:01:56.569155 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:56.559023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44153 (local address 127.0.98.1:33379)
0926 04:01:56.559075 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 04:01:56.559089 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 04:01:56.559152 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:01:56.560030 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:01:56.560033 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:01:56.560439 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:01:56.560621 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:01:56.560673 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:01:56.561746 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:01:56.561747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:01:56.562506 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:01:56.562508 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:01:56.563110 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:01:56.563111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:01:56.563221 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:01:56.563829 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:01:56.563838 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:01:56.565473 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:01:56.567647 (+ 2174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:01:56.567653 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:01:56.567654 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:01:56.567858 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:01:56.568139 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:01:56.568140 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:01:56.568140 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:01:56.568242 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:01:56.568616 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:01:56.568622 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:01:56.568766 (+ 144us) client_negotiation.cc:769] Sending connection context
0926 04:01:56.568883 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 04:01:56.569062 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":6}
I20260926 04:01:56.569167 17290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:56.559159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33379 (local address 127.0.98.3:44153)
0926 04:01:56.559301 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 04:01:56.559312 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 04:01:56.559348 (+ 36us) server_negotiation.cc:407] Connection header received
0926 04:01:56.559790 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:01:56.559792 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:01:56.559850 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:01:56.560001 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:01:56.560786 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:01:56.561624 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:01:56.562614 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:01:56.563003 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:01:56.563143 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:01:56.565587 (+ 2444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:01:56.565591 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:01:56.565591 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:01:56.565600 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:01:56.567562 (+ 1962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:01:56.567970 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:01:56.567973 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:01:56.567974 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:01:56.568063 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:01:56.568315 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:01:56.568316 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:01:56.568317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:01:56.568540 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:01:56.568635 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 04:01:56.568933 (+ 298us) server_negotiation.cc:299] Negotiation successful
0926 04:01:56.569050 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":60,"threads_started":1}
I20260926 04:01:56.569207 17291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:56.559199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53233 (local address 127.0.98.2:41269)
0926 04:01:56.559352 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 04:01:56.559360 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 04:01:56.559384 (+ 24us) server_negotiation.cc:407] Connection header received
0926 04:01:56.559790 (+ 406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:01:56.559792 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:01:56.559837 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:01:56.559994 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:01:56.560771 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:01:56.561623 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:01:56.562603 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:01:56.563001 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:01:56.563138 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:01:56.565942 (+ 2804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:01:56.565945 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:01:56.565945 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:01:56.565956 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:01:56.567757 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:01:56.568118 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:01:56.568121 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:01:56.568121 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:01:56.568215 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:01:56.568425 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:01:56.568427 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:01:56.568427 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:01:56.568696 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:01:56.568777 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 04:01:56.568967 (+ 190us) server_negotiation.cc:299] Negotiation successful
0926 04:01:56.569093 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":70,"threads_started":1}
I20260926 04:01:56.569271 17289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:56.559041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41269 (local address 127.0.98.1:53233)
0926 04:01:56.559170 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 04:01:56.559182 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 04:01:56.559236 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:01:56.560031 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:01:56.560033 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:01:56.560440 (+ 407us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:01:56.560621 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:01:56.560627 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:01:56.561735 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:01:56.561736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:01:56.562507 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:01:56.562509 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:01:56.563112 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:01:56.563114 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:01:56.563222 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:01:56.563878 (+ 656us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:01:56.563885 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:01:56.565843 (+ 1958us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:01:56.567855 (+ 2012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:01:56.567856 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:01:56.567857 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:01:56.568023 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:01:56.568288 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:01:56.568289 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:01:56.568289 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:01:56.568356 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:01:56.568772 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:01:56.568773 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:01:56.568817 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 04:01:56.568908 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 04:01:56.569138 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":169,"thread_start_us":62,"threads_started":1}
W20260926 04:01:56.570585 16904 leader_election.cc:341] T bbb8316c021f4aafb27d2bbdc198ac5f P ffccccc56a0949a29ec497d810c39c43 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 42019c7843344a1983c3fa631a898039 (127.0.98.3:44153): Not found: Tablet not found: bbb8316c021f4aafb27d2bbdc198ac5f
W20260926 04:01:56.570712 16903 leader_election.cc:341] T bbb8316c021f4aafb27d2bbdc198ac5f P ffccccc56a0949a29ec497d810c39c43 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c3b09a5fb725437ca6c07e2fedf6248d (127.0.98.2:41269): Not found: Tablet not found: bbb8316c021f4aafb27d2bbdc198ac5f
I20260926 04:01:56.570773 16903 leader_election.cc:302] T bbb8316c021f4aafb27d2bbdc198ac5f P ffccccc56a0949a29ec497d810c39c43 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ffccccc56a0949a29ec497d810c39c43; no voters: 42019c7843344a1983c3fa631a898039, c3b09a5fb725437ca6c07e2fedf6248d
I20260926 04:01:56.570940 17288 raft_consensus.cc:2726] T bbb8316c021f4aafb27d2bbdc198ac5f P ffccccc56a0949a29ec497d810c39c43 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 04:01:57.077814 17292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:57.077595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32815 (local address 127.0.98.1:46089)
0926 04:01:57.077726 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 04:01:57.077771 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":56,"threads_started":1}
I20260926 04:01:57.139271 17293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:57.139051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32815 (local address 127.0.98.2:59939)
0926 04:01:57.139182 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 04:01:57.139229 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":66,"threads_started":1}
I20260926 04:01:57.280397 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:57.280212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32815 (local address 127.0.98.3:55925)
0926 04:01:57.280325 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0926 04:01:57.280362 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":60,"threads_started":1}
I20260926 04:01:58.078558 17295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:58.078336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32815 (local address 127.0.98.1:37435)
0926 04:01:58.078473 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 04:01:58.078510 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":61,"threads_started":1}
I20260926 04:01:58.140023 17296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:58.139817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32815 (local address 127.0.98.2:54231)
0926 04:01:58.139946 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 04:01:58.139983 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":60,"threads_started":1}
I20260926 04:01:58.230780 17297 raft_consensus.cc:482] T bbb8316c021f4aafb27d2bbdc198ac5f P ffccccc56a0949a29ec497d810c39c43 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 04:01:58.230858 17297 raft_consensus.cc:504] T bbb8316c021f4aafb27d2bbdc198ac5f P ffccccc56a0949a29ec497d810c39c43 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42019c7843344a1983c3fa631a898039" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44153 } } peers { permanent_uuid: "c3b09a5fb725437ca6c07e2fedf6248d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41269 } } peers { permanent_uuid: "ffccccc56a0949a29ec497d810c39c43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39669 } }
I20260926 04:01:58.231082 17297 leader_election.cc:288] T bbb8316c021f4aafb27d2bbdc198ac5f P ffccccc56a0949a29ec497d810c39c43 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42019c7843344a1983c3fa631a898039 (127.0.98.3:44153), c3b09a5fb725437ca6c07e2fedf6248d (127.0.98.2:41269)
W20260926 04:01:58.231510 16903 leader_election.cc:341] T bbb8316c021f4aafb27d2bbdc198ac5f P ffccccc56a0949a29ec497d810c39c43 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c3b09a5fb725437ca6c07e2fedf6248d (127.0.98.2:41269): Not found: Tablet not found: bbb8316c021f4aafb27d2bbdc198ac5f
W20260926 04:01:58.231607 16904 leader_election.cc:341] T bbb8316c021f4aafb27d2bbdc198ac5f P ffccccc56a0949a29ec497d810c39c43 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 42019c7843344a1983c3fa631a898039 (127.0.98.3:44153): Not found: Tablet not found: bbb8316c021f4aafb27d2bbdc198ac5f
I20260926 04:01:58.231721 16904 leader_election.cc:302] T bbb8316c021f4aafb27d2bbdc198ac5f P ffccccc56a0949a29ec497d810c39c43 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ffccccc56a0949a29ec497d810c39c43; no voters: 42019c7843344a1983c3fa631a898039, c3b09a5fb725437ca6c07e2fedf6248d
I20260926 04:01:58.231817 17297 raft_consensus.cc:2726] T bbb8316c021f4aafb27d2bbdc198ac5f P ffccccc56a0949a29ec497d810c39c43 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 04:01:58.281101 17298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:58.280895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32815 (local address 127.0.98.3:36005)
0926 04:01:58.281025 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 04:01:58.281060 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":61,"threads_started":1}
I20260926 04:01:59.079280 17299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:59.079064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32815 (local address 127.0.98.1:38069)
0926 04:01:59.079202 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 04:01:59.079235 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":74,"threads_started":1}
I20260926 04:01:59.140695 17300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:59.140486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32815 (local address 127.0.98.2:36929)
0926 04:01:59.140588 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0926 04:01:59.140620 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":56,"threads_started":1}
I20260926 04:01:59.281769 17301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:01:59.281570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32815 (local address 127.0.98.3:38559)
0926 04:01:59.281698 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 04:01:59.281731 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":59,"threads_started":1}
I20260926 04:01:59.806205 17302 raft_consensus.cc:482] T bbb8316c021f4aafb27d2bbdc198ac5f P ffccccc56a0949a29ec497d810c39c43 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 04:01:59.806326 17302 raft_consensus.cc:504] T bbb8316c021f4aafb27d2bbdc198ac5f P ffccccc56a0949a29ec497d810c39c43 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42019c7843344a1983c3fa631a898039" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44153 } } peers { permanent_uuid: "c3b09a5fb725437ca6c07e2fedf6248d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41269 } } peers { permanent_uuid: "ffccccc56a0949a29ec497d810c39c43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39669 } }
I20260926 04:01:59.806535 17302 leader_election.cc:288] T bbb8316c021f4aafb27d2bbdc198ac5f P ffccccc56a0949a29ec497d810c39c43 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42019c7843344a1983c3fa631a898039 (127.0.98.3:44153), c3b09a5fb725437ca6c07e2fedf6248d (127.0.98.2:41269)
W20260926 04:01:59.806883 16903 leader_election.cc:341] T bbb8316c021f4aafb27d2bbdc198ac5f P ffccccc56a0949a29ec497d810c39c43 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c3b09a5fb725437ca6c07e2fedf6248d (127.0.98.2:41269): Not found: Tablet not found: bbb8316c021f4aafb27d2bbdc198ac5f
W20260926 04:01:59.806957 16904 leader_election.cc:341] T bbb8316c021f4aafb27d2bbdc198ac5f P ffccccc56a0949a29ec497d810c39c43 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 42019c7843344a1983c3fa631a898039 (127.0.98.3:44153): Not found: Tablet not found: bbb8316c021f4aafb27d2bbdc198ac5f
I20260926 04:01:59.807017 16904 leader_election.cc:302] T bbb8316c021f4aafb27d2bbdc198ac5f P ffccccc56a0949a29ec497d810c39c43 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ffccccc56a0949a29ec497d810c39c43; no voters: 42019c7843344a1983c3fa631a898039, c3b09a5fb725437ca6c07e2fedf6248d
I20260926 04:01:59.807102 17302 raft_consensus.cc:2726] T bbb8316c021f4aafb27d2bbdc198ac5f P ffccccc56a0949a29ec497d810c39c43 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 04:02:00.079948 17303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:00.079756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32815 (local address 127.0.98.1:59013)
0926 04:02:00.079871 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:00.079902 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":58,"threads_started":1}
I20260926 04:02:00.141322 17304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:00.141126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32815 (local address 127.0.98.2:33297)
0926 04:02:00.141247 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:00.141283 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":57,"threads_started":1}
I20260926 04:02:00.282328 17305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:00.282133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32815 (local address 127.0.98.3:55477)
0926 04:02:00.282258 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:00.282293 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":54,"threads_started":1}
I20260926 04:02:01.080732 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:01.080488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32815 (local address 127.0.98.1:44713)
0926 04:02:01.080613 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:01.080686 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":63,"threads_started":1}
I20260926 04:02:01.141891 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:01.141702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32815 (local address 127.0.98.2:44501)
0926 04:02:01.141820 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:01.141854 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":63,"threads_started":1}
I20260926 04:02:01.282985 17308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:01.282767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32815 (local address 127.0.98.3:36169)
0926 04:02:01.282912 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:01.282945 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":93,"threads_started":1}
I20260926 04:02:02.081441 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:02.081192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32815 (local address 127.0.98.1:41503)
0926 04:02:02.081335 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:02.081384 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":92,"threads_started":1}
I20260926 04:02:02.142558 17310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:02.142346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32815 (local address 127.0.98.2:41369)
0926 04:02:02.142483 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:02.142519 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":57,"threads_started":1}
I20260926 04:02:02.283589 17311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:02.283393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32815 (local address 127.0.98.3:51847)
0926 04:02:02.283515 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:02.283551 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":60,"threads_started":1}
I20260926 04:02:03.082132 17312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:03.081910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32815 (local address 127.0.98.1:46541)
0926 04:02:03.082050 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:03.082088 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":92,"threads_started":1}
I20260926 04:02:03.143217 17313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:03.143019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32815 (local address 127.0.98.2:41237)
0926 04:02:03.143139 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:03.143176 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":60,"threads_started":1}
I20260926 04:02:03.284304 17314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:03.284065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32815 (local address 127.0.98.3:33767)
0926 04:02:03.284195 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:03.284238 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":57,"threads_started":1}
I20260926 04:02:03.987849 17315 raft_consensus.cc:482] T bbb8316c021f4aafb27d2bbdc198ac5f P ffccccc56a0949a29ec497d810c39c43 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 04:02:03.987965 17315 raft_consensus.cc:504] T bbb8316c021f4aafb27d2bbdc198ac5f P ffccccc56a0949a29ec497d810c39c43 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42019c7843344a1983c3fa631a898039" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44153 } } peers { permanent_uuid: "c3b09a5fb725437ca6c07e2fedf6248d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41269 } } peers { permanent_uuid: "ffccccc56a0949a29ec497d810c39c43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39669 } }
I20260926 04:02:03.988168 17315 leader_election.cc:288] T bbb8316c021f4aafb27d2bbdc198ac5f P ffccccc56a0949a29ec497d810c39c43 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42019c7843344a1983c3fa631a898039 (127.0.98.3:44153), c3b09a5fb725437ca6c07e2fedf6248d (127.0.98.2:41269)
W20260926 04:02:03.988512 16904 leader_election.cc:341] T bbb8316c021f4aafb27d2bbdc198ac5f P ffccccc56a0949a29ec497d810c39c43 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 42019c7843344a1983c3fa631a898039 (127.0.98.3:44153): Not found: Tablet not found: bbb8316c021f4aafb27d2bbdc198ac5f
W20260926 04:02:03.988597 16903 leader_election.cc:341] T bbb8316c021f4aafb27d2bbdc198ac5f P ffccccc56a0949a29ec497d810c39c43 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c3b09a5fb725437ca6c07e2fedf6248d (127.0.98.2:41269): Not found: Tablet not found: bbb8316c021f4aafb27d2bbdc198ac5f
I20260926 04:02:03.988704 16903 leader_election.cc:302] T bbb8316c021f4aafb27d2bbdc198ac5f P ffccccc56a0949a29ec497d810c39c43 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ffccccc56a0949a29ec497d810c39c43; no voters: 42019c7843344a1983c3fa631a898039, c3b09a5fb725437ca6c07e2fedf6248d
I20260926 04:02:03.988811 17315 raft_consensus.cc:2726] T bbb8316c021f4aafb27d2bbdc198ac5f P ffccccc56a0949a29ec497d810c39c43 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 04:02:04.082904 17316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:04.082662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32815 (local address 127.0.98.1:46673)
0926 04:02:04.082806 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:04.082853 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":64,"threads_started":1}
I20260926 04:02:04.143989 17317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:04.143727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32815 (local address 127.0.98.2:48763)
0926 04:02:04.143890 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:04.143930 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":76,"threads_started":1}
I20260926 04:02:04.285254 17318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:04.284944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32815 (local address 127.0.98.3:48165)
0926 04:02:04.285148 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:04.285193 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":111,"threads_started":1}
I20260926 04:02:05.083623 17319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:05.083385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32815 (local address 127.0.98.1:50475)
0926 04:02:05.083525 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:05.083565 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":67,"threads_started":1}
I20260926 04:02:05.144735 17320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:05.144490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32815 (local address 127.0.98.2:51195)
0926 04:02:05.144614 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:05.144694 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":58,"threads_started":1}
I20260926 04:02:05.285892 17321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:05.285684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32815 (local address 127.0.98.3:42411)
0926 04:02:05.285809 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:05.285850 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":83,"threads_started":1}
I20260926 04:02:06.084348 17322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:06.084130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32815 (local address 127.0.98.1:49231)
0926 04:02:06.084268 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:06.084302 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":71,"threads_started":1}
I20260926 04:02:06.145524 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:06.145308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32815 (local address 127.0.98.2:50019)
0926 04:02:06.145446 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:06.145484 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":73,"threads_started":1}
I20260926 04:02:06.286616 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:06.286394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32815 (local address 127.0.98.3:37867)
0926 04:02:06.286529 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:06.286567 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":65,"threads_started":1}
I20260926 04:02:07.085002 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:07.084753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32815 (local address 127.0.98.1:38457)
0926 04:02:07.084918 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:07.084954 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":62,"threads_started":1}
I20260926 04:02:07.092401 17326 raft_consensus.cc:482] T bbb8316c021f4aafb27d2bbdc198ac5f P ffccccc56a0949a29ec497d810c39c43 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 04:02:07.092478 17326 raft_consensus.cc:504] T bbb8316c021f4aafb27d2bbdc198ac5f P ffccccc56a0949a29ec497d810c39c43 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42019c7843344a1983c3fa631a898039" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44153 } } peers { permanent_uuid: "c3b09a5fb725437ca6c07e2fedf6248d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41269 } } peers { permanent_uuid: "ffccccc56a0949a29ec497d810c39c43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39669 } }
I20260926 04:02:07.092676 17326 leader_election.cc:288] T bbb8316c021f4aafb27d2bbdc198ac5f P ffccccc56a0949a29ec497d810c39c43 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42019c7843344a1983c3fa631a898039 (127.0.98.3:44153), c3b09a5fb725437ca6c07e2fedf6248d (127.0.98.2:41269)
W20260926 04:02:07.093063 16903 leader_election.cc:341] T bbb8316c021f4aafb27d2bbdc198ac5f P ffccccc56a0949a29ec497d810c39c43 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c3b09a5fb725437ca6c07e2fedf6248d (127.0.98.2:41269): Not found: Tablet not found: bbb8316c021f4aafb27d2bbdc198ac5f
W20260926 04:02:07.093158 16904 leader_election.cc:341] T bbb8316c021f4aafb27d2bbdc198ac5f P ffccccc56a0949a29ec497d810c39c43 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 42019c7843344a1983c3fa631a898039 (127.0.98.3:44153): Not found: Tablet not found: bbb8316c021f4aafb27d2bbdc198ac5f
I20260926 04:02:07.093194 16904 leader_election.cc:302] T bbb8316c021f4aafb27d2bbdc198ac5f P ffccccc56a0949a29ec497d810c39c43 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ffccccc56a0949a29ec497d810c39c43; no voters: 42019c7843344a1983c3fa631a898039, c3b09a5fb725437ca6c07e2fedf6248d
I20260926 04:02:07.093282 17326 raft_consensus.cc:2726] T bbb8316c021f4aafb27d2bbdc198ac5f P ffccccc56a0949a29ec497d810c39c43 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 04:02:07.146286 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:07.146077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32815 (local address 127.0.98.2:51921)
0926 04:02:07.146203 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:07.146242 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":60,"threads_started":1}
I20260926 04:02:07.287334 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:07.287131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32815 (local address 127.0.98.3:34849)
0926 04:02:07.287246 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:07.287279 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":55,"threads_started":1}
I20260926 04:02:08.085645 17329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:08.085443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32815 (local address 127.0.98.1:52007)
0926 04:02:08.085564 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:08.085600 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":63,"threads_started":1}
I20260926 04:02:08.146957 17330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:08.146711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32815 (local address 127.0.98.2:58569)
0926 04:02:08.146833 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:08.146913 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":58,"threads_started":1}
I20260926 04:02:08.287961 17331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:08.287760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32815 (local address 127.0.98.3:55871)
0926 04:02:08.287885 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:08.287918 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":58,"threads_started":1}
I20260926 04:02:09.086313 17332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:09.086101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32815 (local address 127.0.98.1:39293)
0926 04:02:09.086227 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:09.086264 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":63,"threads_started":1}
I20260926 04:02:09.147650 17333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:09.147445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32815 (local address 127.0.98.2:54249)
0926 04:02:09.147570 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:09.147607 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":60,"threads_started":1}
I20260926 04:02:09.288743 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:09.288450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32815 (local address 127.0.98.3:47479)
0926 04:02:09.288593 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:09.288683 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":77,"threads_started":1}
I20260926 04:02:10.087037 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:10.086835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32815 (local address 127.0.98.1:41885)
0926 04:02:10.086956 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:10.086989 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":62,"threads_started":1}
I20260926 04:02:10.148370 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:10.148169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32815 (local address 127.0.98.2:40557)
0926 04:02:10.148297 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:10.148326 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":71,"threads_started":1}
I20260926 04:02:10.289477 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:10.289250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32815 (local address 127.0.98.3:42151)
0926 04:02:10.289393 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:10.289428 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:32815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":74,"threads_started":1}
I20260926 04:02:10.294031 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 16851
I20260926 04:02:10.298820 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 17019
I20260926 04:02:10.303279 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 17152
2026-09-26T04:02:10Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58451 ms)
[----------] 30 tests from SecurityITest (234835 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 04:02:10 dist-test-slave-nhpw krb5kdc[17340](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 04:02:10 dist-test-slave-nhpw krb5kdc[17340](info): set up 2 sockets
Sep 26 04:02:10 dist-test-slave-nhpw krb5kdc[17340](info): commencing operation
krb5kdc: starting...
W20260926 04:02:11.340255 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/krb5kdc/test-user.keytab.
Sep 26 04:02:11 dist-test-slave-nhpw krb5kdc[17340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T04:02:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:02:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 04:02:11.456228 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35803
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35803
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:02:11.535432 17353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:02:11.535643 17353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:02:11.535665 17353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:02:11.536871 17353 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:02:11.536916 17353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:02:11.536931 17353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:02:11.536944 17353 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:02:11.536958 17353 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:02:11.538451 17353 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35803
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:02:11.538698 17353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:02:11.538934 17353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:02:11.543982 17359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:02:11.543982 17361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:02:11.544081 17358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:02:11.544185 17353 server_base.cc:1034] running on GCE node
I20260926 04:02:11.544423 17353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:02:11.544804 17353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:02:11.545990 17353 hybrid_clock.cc:648] HybridClock initialized: now 1790395331545973 us; error 34 us; skew 500 ppm
Sep 26 04:02:11 dist-test-slave-nhpw krb5kdc[17340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395331, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:02:11.548413 17353 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 04:02:11.548909 17353 webserver.cc:460] Webserver started at http://127.0.98.62:35945/ using document root <none> and password file <none>
I20260926 04:02:11.549150 17353 fs_manager.cc:359] Metadata directory not provided
I20260926 04:02:11.549216 17353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:02:11.549348 17353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:02:11.550382 17353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/minicluster-data/master-0/data/instance:
uuid: "2f1181ce31be4563a3ef730c7a5fa4f9"
format_stamp: "Formatted at 2026-09-26 04:02:11 on dist-test-slave-nhpw"
server_key: "27e94c236db995e5ae6b69946b746c15"
server_key_iv: "1f3dc7b1e4680f5997f86691d753ecfe"
server_key_version: "encryptionkey@0"
I20260926 04:02:11.550751 17353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/minicluster-data/master-0/wal/instance:
uuid: "2f1181ce31be4563a3ef730c7a5fa4f9"
format_stamp: "Formatted at 2026-09-26 04:02:11 on dist-test-slave-nhpw"
server_key: "27e94c236db995e5ae6b69946b746c15"
server_key_iv: "1f3dc7b1e4680f5997f86691d753ecfe"
server_key_version: "encryptionkey@0"
I20260926 04:02:11.552052 17353 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 04:02:11.552811 17368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:11.552995 17353 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:02:11.553092 17353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/minicluster-data/master-0/wal
uuid: "2f1181ce31be4563a3ef730c7a5fa4f9"
format_stamp: "Formatted at 2026-09-26 04:02:11 on dist-test-slave-nhpw"
server_key: "27e94c236db995e5ae6b69946b746c15"
server_key_iv: "1f3dc7b1e4680f5997f86691d753ecfe"
server_key_version: "encryptionkey@0"
I20260926 04:02:11.553177 17353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:02:11.560475 17353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:02:11.562431 17353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:02:11.562604 17353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:02:11.566764 17353 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35803
I20260926 04:02:11.566799 17420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35803 every 8 connection(s)
I20260926 04:02:11.567131 17353 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
I20260926 04:02:11.567932 17421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:02:11.570144 17421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f1181ce31be4563a3ef730c7a5fa4f9: Bootstrap starting.
I20260926 04:02:11.570648 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 17353
I20260926 04:02:11.570762 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/minicluster-data/master-0/wal/instance
I20260926 04:02:11.570796 17421 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f1181ce31be4563a3ef730c7a5fa4f9: Neither blocks nor log segments found. Creating new log.
I20260926 04:02:11.570976 392 external_mini_cluster.cc:1428] Setting key 0dc366094793bfcf844143be415e463f
I20260926 04:02:11.571123 17421 log.cc:824] T 00000000000000000000000000000000 P 2f1181ce31be4563a3ef730c7a5fa4f9: Log is configured to *not* fsync() on all Append() calls
I20260926 04:02:11.571908 17421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f1181ce31be4563a3ef730c7a5fa4f9: No bootstrap required, opened a new log
I20260926 04:02:11.574218 17421 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f1181ce31be4563a3ef730c7a5fa4f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f1181ce31be4563a3ef730c7a5fa4f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35803 } }
I20260926 04:02:11.574389 17421 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f1181ce31be4563a3ef730c7a5fa4f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:02:11.574425 17421 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f1181ce31be4563a3ef730c7a5fa4f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f1181ce31be4563a3ef730c7a5fa4f9, State: Initialized, Role: FOLLOWER
I20260926 04:02:11.574542 17421 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f1181ce31be4563a3ef730c7a5fa4f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f1181ce31be4563a3ef730c7a5fa4f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35803 } }
I20260926 04:02:11.574620 17421 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f1181ce31be4563a3ef730c7a5fa4f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:02:11.574649 17421 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f1181ce31be4563a3ef730c7a5fa4f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:02:11.574693 17421 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f1181ce31be4563a3ef730c7a5fa4f9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:02:11.575344 17421 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f1181ce31be4563a3ef730c7a5fa4f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f1181ce31be4563a3ef730c7a5fa4f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35803 } }
I20260926 04:02:11.575471 17421 leader_election.cc:302] T 00000000000000000000000000000000 P 2f1181ce31be4563a3ef730c7a5fa4f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f1181ce31be4563a3ef730c7a5fa4f9; no voters:
I20260926 04:02:11.575634 17421 leader_election.cc:288] T 00000000000000000000000000000000 P 2f1181ce31be4563a3ef730c7a5fa4f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 04:02:11.575680 17426 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f1181ce31be4563a3ef730c7a5fa4f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:02:11.575837 17426 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f1181ce31be4563a3ef730c7a5fa4f9 [term 1 LEADER]: Becoming Leader. State: Replica: 2f1181ce31be4563a3ef730c7a5fa4f9, State: Running, Role: LEADER
I20260926 04:02:11.575886 17421 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f1181ce31be4563a3ef730c7a5fa4f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:02:11.576004 17426 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f1181ce31be4563a3ef730c7a5fa4f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f1181ce31be4563a3ef730c7a5fa4f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35803 } }
I20260926 04:02:11.576383 17427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f1181ce31be4563a3ef730c7a5fa4f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f1181ce31be4563a3ef730c7a5fa4f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f1181ce31be4563a3ef730c7a5fa4f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35803 } } }
I20260926 04:02:11.576478 17427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f1181ce31be4563a3ef730c7a5fa4f9 [sys.catalog]: This master's current role is: LEADER
I20260926 04:02:11.576819 17434 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:02:11.577342 17434 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:02:11.578090 17428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f1181ce31be4563a3ef730c7a5fa4f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f1181ce31be4563a3ef730c7a5fa4f9. Latest consensus state: current_term: 1 leader_uuid: "2f1181ce31be4563a3ef730c7a5fa4f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f1181ce31be4563a3ef730c7a5fa4f9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35803 } } }
I20260926 04:02:11.578243 17428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f1181ce31be4563a3ef730c7a5fa4f9 [sys.catalog]: This master's current role is: LEADER
Sep 26 04:02:11 dist-test-slave-nhpw krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 04:02:11.579543 17434 catalog_manager.cc:1347] Generated new cluster ID: 6a14f2894c3f4a32b5d520d26f122795
I20260926 04:02:11.579600 17434 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 04:02:11.582278 17425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:11.571510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60758 (local address 127.0.98.62:35803)
0926 04:02:11.571710 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:11.571717 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:11.571740 (+ 23us) server_negotiation.cc:407] Connection header received
0926 04:02:11.572207 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:11.572214 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:11.572262 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:11.572371 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:11.573202 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:11.574094 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:11.575178 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:11.575674 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:11.576122 (+ 448us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:11.579071 (+ 2949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:11.579075 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:11.579075 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:02:11.579085 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:11.580875 (+ 1790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:11.581239 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:11.581241 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:11.581241 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:11.581322 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:11.581513 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:11.581514 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:11.581514 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:11.581725 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:11.581823 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:11.581935 (+ 112us) server_negotiation.cc:299] Negotiation successful
0926 04:02:11.582043 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":98,"threads_started":1}
I20260926 04:02:11.593035 17434 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:02:11.593590 17434 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:02:11.600023 17434 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f1181ce31be4563a3ef730c7a5fa4f9: Generated new TSK 0
I20260926 04:02:11.600215 17434 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 04:02:11.627873 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35803
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:02:11.707778 17447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:02:11.707969 17447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:02:11.707993 17447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:02:11.709190 17447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:02:11.709230 17447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:02:11.709256 17447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 04:02:11.710695 17447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:02:11.710896 17447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:02:11.711100 17447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:02:11.715850 17453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:02:11.715890 17452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:02:11.715866 17455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:02:11.716097 17447 server_base.cc:1034] running on GCE node
I20260926 04:02:11.716285 17447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:02:11.716538 17447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:02:11.717700 17447 hybrid_clock.cc:648] HybridClock initialized: now 1790395331717688 us; error 40 us; skew 500 ppm
Sep 26 04:02:11 dist-test-slave-nhpw krb5kdc[17340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395331, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:02:11.720104 17447 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 04:02:11.720562 17447 webserver.cc:460] Webserver started at http://127.0.98.1:41903/ using document root <none> and password file <none>
I20260926 04:02:11.720839 17447 fs_manager.cc:359] Metadata directory not provided
I20260926 04:02:11.720907 17447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:02:11.721046 17447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:02:11.722086 17447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/minicluster-data/ts-0/data/instance:
uuid: "0b85abd838e748acb7130d4327e4e6aa"
format_stamp: "Formatted at 2026-09-26 04:02:11 on dist-test-slave-nhpw"
server_key: "e8ac058ee6f6ebcfed19ebb2453ef867"
server_key_iv: "c8ac6a59417fa8fd3925f6c9be4e996e"
server_key_version: "encryptionkey@0"
I20260926 04:02:11.722450 17447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0b85abd838e748acb7130d4327e4e6aa"
format_stamp: "Formatted at 2026-09-26 04:02:11 on dist-test-slave-nhpw"
server_key: "e8ac058ee6f6ebcfed19ebb2453ef867"
server_key_iv: "c8ac6a59417fa8fd3925f6c9be4e996e"
server_key_version: "encryptionkey@0"
I20260926 04:02:11.723843 17447 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 04:02:11.724565 17462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:11.724768 17447 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:02:11.724838 17447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/minicluster-data/ts-0/wal
uuid: "0b85abd838e748acb7130d4327e4e6aa"
format_stamp: "Formatted at 2026-09-26 04:02:11 on dist-test-slave-nhpw"
server_key: "e8ac058ee6f6ebcfed19ebb2453ef867"
server_key_iv: "c8ac6a59417fa8fd3925f6c9be4e996e"
server_key_version: "encryptionkey@0"
I20260926 04:02:11.724922 17447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:02:11.737648 17447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:02:11.739882 17447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:02:11.740059 17447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:02:11.740283 17447 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:02:11.740593 17447 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:02:11.740697 17447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:11.740752 17447 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:02:11.740780 17447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:11.746721 17447 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45373
I20260926 04:02:11.746773 17575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45373 every 8 connection(s)
I20260926 04:02:11.747108 17447 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
I20260926 04:02:11.752197 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 17447
I20260926 04:02:11.752322 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/minicluster-data/ts-0/wal/instance
I20260926 04:02:11.752547 392 external_mini_cluster.cc:1428] Setting key c2862fa4ccdcc1e5c733c1986f14d24d
Sep 26 04:02:11 dist-test-slave-nhpw krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395331, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 04:02:11.757258 17425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:11.747883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52993 (local address 127.0.98.62:35803)
0926 04:02:11.747972 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:11.747974 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:11.748168 (+ 194us) server_negotiation.cc:407] Connection header received
0926 04:02:11.749266 (+ 1098us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:11.749267 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:11.749281 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:11.749327 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:11.749970 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:11.750489 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:11.751546 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:11.751774 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:11.751863 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:11.754227 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:11.754228 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:11.754229 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:02:11.754238 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:11.755722 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:11.756164 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:11.756165 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:11.756166 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:11.756226 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:11.756556 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:11.756557 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:11.756558 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:11.756739 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:11.756814 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:11.757084 (+ 270us) server_negotiation.cc:299] Negotiation successful
0926 04:02:11.757163 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 04:02:11.757465 17578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:11.747784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35803 (local address 127.0.98.1:52993)
0926 04:02:11.748033 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:11.748080 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 04:02:11.749130 (+ 1050us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:02:11.749410 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:02:11.749414 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:02:11.749664 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:02:11.749851 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:11.749855 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:11.750588 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:11.750589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:11.751436 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:11.751438 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:11.751874 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:11.751875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:11.752001 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:11.752638 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:02:11.752676 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:02:11.754131 (+ 1455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:02:11.755819 (+ 1688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:02:11.755827 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:02:11.755829 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:02:11.756053 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:02:11.756310 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:02:11.756311 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:02:11.756311 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:02:11.756465 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:02:11.756806 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:02:11.756813 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:02:11.756956 (+ 143us) client_negotiation.cc:769] Sending connection context
0926 04:02:11.757066 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 04:02:11.757232 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":112,"threads_started":1}
I20260926 04:02:11.757977 17576 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35803
I20260926 04:02:11.758068 17576 heartbeater.cc:464] Registering TS with master...
I20260926 04:02:11.758263 17576 heartbeater.cc:511] Master 127.0.98.62:35803 requested a full tablet report, sending...
I20260926 04:02:11.758832 17385 ts_manager.cc:194] Registered new tserver with Master: 0b85abd838e748acb7130d4327e4e6aa (127.0.98.1:45373)
I20260926 04:02:11.759807 17385 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52993
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 04:02:11.776587 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35803
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:02:11.855219 17581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:02:11.855415 17581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:02:11.855439 17581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:02:11.856737 17581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:02:11.856777 17581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:02:11.856803 17581 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 04:02:11.858260 17581 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:02:11.858492 17581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:02:11.858727 17581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:02:11.863567 17589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:02:11.863592 17586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:02:11.863703 17581 server_base.cc:1034] running on GCE node
W20260926 04:02:11.863567 17587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:02:11.864104 17581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:02:11.864333 17581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:02:11.865505 17581 hybrid_clock.cc:648] HybridClock initialized: now 1790395331865492 us; error 28 us; skew 500 ppm
Sep 26 04:02:11 dist-test-slave-nhpw krb5kdc[17340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395331, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:02:11.868328 17581 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 04:02:11.868852 17581 webserver.cc:460] Webserver started at http://127.0.98.2:34701/ using document root <none> and password file <none>
I20260926 04:02:11.869067 17581 fs_manager.cc:359] Metadata directory not provided
I20260926 04:02:11.869132 17581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:02:11.869287 17581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:02:11.870297 17581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/minicluster-data/ts-1/data/instance:
uuid: "ddaa913c1bb44369a6d4f397848918b8"
format_stamp: "Formatted at 2026-09-26 04:02:11 on dist-test-slave-nhpw"
server_key: "0a3042ae3711ba8ba6f7cb20c79fc8bb"
server_key_iv: "cf959bc44da47c76e546bba51903469a"
server_key_version: "encryptionkey@0"
I20260926 04:02:11.870657 17581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ddaa913c1bb44369a6d4f397848918b8"
format_stamp: "Formatted at 2026-09-26 04:02:11 on dist-test-slave-nhpw"
server_key: "0a3042ae3711ba8ba6f7cb20c79fc8bb"
server_key_iv: "cf959bc44da47c76e546bba51903469a"
server_key_version: "encryptionkey@0"
I20260926 04:02:11.871937 17581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 04:02:11.872730 17596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:11.872912 17581 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:02:11.873006 17581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/minicluster-data/ts-1/wal
uuid: "ddaa913c1bb44369a6d4f397848918b8"
format_stamp: "Formatted at 2026-09-26 04:02:11 on dist-test-slave-nhpw"
server_key: "0a3042ae3711ba8ba6f7cb20c79fc8bb"
server_key_iv: "cf959bc44da47c76e546bba51903469a"
server_key_version: "encryptionkey@0"
I20260926 04:02:11.873090 17581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:02:11.893026 17581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:02:11.894932 17581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:02:11.895109 17581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:02:11.895345 17581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:02:11.895650 17581 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:02:11.895705 17581 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:11.895746 17581 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:02:11.895776 17581 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:11.901669 17581 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44367
I20260926 04:02:11.901716 17709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44367 every 8 connection(s)
I20260926 04:02:11.902012 17581 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
Sep 26 04:02:11 dist-test-slave-nhpw krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395331, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 04:02:11.910636 17425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:11.902677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33251 (local address 127.0.98.62:35803)
0926 04:02:11.902750 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:11.902752 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:11.902977 (+ 225us) server_negotiation.cc:407] Connection header received
0926 04:02:11.903842 (+ 865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:11.903843 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:11.903853 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:11.903901 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:11.904463 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:11.905045 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:11.905978 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:11.906156 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:11.906208 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:11.908179 (+ 1971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:11.908180 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:11.908180 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:02:11.908187 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:11.909574 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:11.909907 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:11.909907 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:11.909908 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:11.909957 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:11.910162 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:11.910162 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:11.910162 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:11.910252 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:11.910301 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:11.910514 (+ 213us) server_negotiation.cc:299] Negotiation successful
0926 04:02:11.910573 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 04:02:11.910780 17712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:11.902652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35803 (local address 127.0.98.2:33251)
0926 04:02:11.902842 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:11.902889 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 04:02:11.903753 (+ 864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:02:11.903961 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:02:11.903964 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:02:11.904195 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:02:11.904381 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:11.904385 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:11.905122 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:11.905123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:11.905902 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:11.905904 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:11.906225 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:11.906226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:11.906319 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:11.906729 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:02:11.906737 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:02:11.908093 (+ 1356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:02:11.909639 (+ 1546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:02:11.909643 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:02:11.909645 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:02:11.909841 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:02:11.910019 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:02:11.910019 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:02:11.910019 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:02:11.910109 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:02:11.910305 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:02:11.910309 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:02:11.910412 (+ 103us) client_negotiation.cc:769] Sending connection context
0926 04:02:11.910489 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 04:02:11.910596 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":112,"threads_started":1}
I20260926 04:02:11.911029 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 17581
I20260926 04:02:11.911163 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/minicluster-data/ts-1/wal/instance
I20260926 04:02:11.911314 17710 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35803
I20260926 04:02:11.911389 17710 heartbeater.cc:464] Registering TS with master...
I20260926 04:02:11.911422 392 external_mini_cluster.cc:1428] Setting key 201a68841d3b90a18cdde10aedb5e291
I20260926 04:02:11.911579 17710 heartbeater.cc:511] Master 127.0.98.62:35803 requested a full tablet report, sending...
I20260926 04:02:11.911945 17384 ts_manager.cc:194] Registered new tserver with Master: ddaa913c1bb44369a6d4f397848918b8 (127.0.98.2:44367)
I20260926 04:02:11.912320 17384 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33251
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 04:02:11.935359 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35803
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44433
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:02:12.016387 17715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:02:12.016579 17715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:02:12.016601 17715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:02:12.017774 17715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:02:12.017817 17715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:02:12.017890 17715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 04:02:12.019340 17715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44433
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35803
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:02:12.019562 17715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:02:12.019770 17715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:02:12.024680 17723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:02:12.024675 17720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:02:12.024701 17715 server_base.cc:1034] running on GCE node
W20260926 04:02:12.024675 17721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:02:12.025098 17715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:02:12.025280 17715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:02:12.026414 17715 hybrid_clock.cc:648] HybridClock initialized: now 1790395332026402 us; error 28 us; skew 500 ppm
Sep 26 04:02:12 dist-test-slave-nhpw krb5kdc[17340](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395332, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:02:12.028924 17715 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 04:02:12.029359 17715 webserver.cc:460] Webserver started at http://127.0.98.3:44239/ using document root <none> and password file <none>
I20260926 04:02:12.029572 17715 fs_manager.cc:359] Metadata directory not provided
I20260926 04:02:12.029614 17715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:02:12.029773 17715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:02:12.030743 17715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/minicluster-data/ts-2/data/instance:
uuid: "22ab4c101f2340978ee9395bad1f4d4c"
format_stamp: "Formatted at 2026-09-26 04:02:12 on dist-test-slave-nhpw"
server_key: "d5ea10bd53b0e8cf9b685f574754e693"
server_key_iv: "d2bcd4aa9ad66aee0988835b53636012"
server_key_version: "encryptionkey@0"
I20260926 04:02:12.031086 17715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/minicluster-data/ts-2/wal/instance:
uuid: "22ab4c101f2340978ee9395bad1f4d4c"
format_stamp: "Formatted at 2026-09-26 04:02:12 on dist-test-slave-nhpw"
server_key: "d5ea10bd53b0e8cf9b685f574754e693"
server_key_iv: "d2bcd4aa9ad66aee0988835b53636012"
server_key_version: "encryptionkey@0"
I20260926 04:02:12.032321 17715 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 04:02:12.033068 17730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:12.033248 17715 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:12.033315 17715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/minicluster-data/ts-2/wal
uuid: "22ab4c101f2340978ee9395bad1f4d4c"
format_stamp: "Formatted at 2026-09-26 04:02:12 on dist-test-slave-nhpw"
server_key: "d5ea10bd53b0e8cf9b685f574754e693"
server_key_iv: "d2bcd4aa9ad66aee0988835b53636012"
server_key_version: "encryptionkey@0"
I20260926 04:02:12.033401 17715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:02:12.066290 17715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:02:12.068507 17715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:02:12.068708 17715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:02:12.068943 17715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:02:12.069269 17715 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:02:12.069300 17715 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:12.069351 17715 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:02:12.069376 17715 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:12.075223 17715 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43963
I20260926 04:02:12.075266 17843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43963 every 8 connection(s)
I20260926 04:02:12.075629 17715 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
I20260926 04:02:12.079834 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 17715
I20260926 04:02:12.079954 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/minicluster-data/ts-2/wal/instance
I20260926 04:02:12.080152 392 external_mini_cluster.cc:1428] Setting key ffc03a97799ac2e5b142757d6d7eccb9
Sep 26 04:02:12 dist-test-slave-nhpw krb5kdc[17340](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395332, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 04:02:12.085197 17425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:12.076373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40975 (local address 127.0.98.62:35803)
0926 04:02:12.076449 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:12.076452 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:12.076620 (+ 168us) server_negotiation.cc:407] Connection header received
0926 04:02:12.077790 (+ 1170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:12.077791 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:12.077804 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:12.077872 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:12.078465 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:12.078965 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:12.080258 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:12.080537 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:12.080623 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:12.082818 (+ 2195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:12.082819 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:12.082820 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:02:12.082826 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:12.084017 (+ 1191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:12.084395 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:12.084396 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:12.084396 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:12.084443 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:12.084624 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:12.084625 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:12.084625 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:12.084776 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:12.084829 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:12.085064 (+ 235us) server_negotiation.cc:299] Negotiation successful
0926 04:02:12.085132 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 04:02:12.085363 17846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:12.076334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35803 (local address 127.0.98.3:40975)
0926 04:02:12.076488 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:12.076543 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 04:02:12.077683 (+ 1140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:02:12.077940 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:02:12.077944 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:02:12.078190 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:02:12.078367 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:12.078370 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:12.079081 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:12.079082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:12.080158 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:12.080160 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:12.080669 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:12.080670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:12.080816 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:12.081409 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:02:12.081417 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:02:12.082732 (+ 1315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:02:12.084106 (+ 1374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:02:12.084111 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:02:12.084113 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:02:12.084330 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:02:12.084485 (+ 155us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:02:12.084485 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:02:12.084485 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:02:12.084573 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:02:12.084835 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:02:12.084840 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:02:12.084949 (+ 109us) client_negotiation.cc:769] Sending connection context
0926 04:02:12.085035 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 04:02:12.085146 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":86,"threads_started":1}
I20260926 04:02:12.085800 17844 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35803
I20260926 04:02:12.085873 17844 heartbeater.cc:464] Registering TS with master...
I20260926 04:02:12.086084 17844 heartbeater.cc:511] Master 127.0.98.62:35803 requested a full tablet report, sending...
I20260926 04:02:12.086438 17384 ts_manager.cc:194] Registered new tserver with Master: 22ab4c101f2340978ee9395bad1f4d4c (127.0.98.3:43963)
I20260926 04:02:12.086807 17384 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40975
I20260926 04:02:12.091805 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 04:02:12.099263 17425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:12.093159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60762 (local address 127.0.98.62:35803)
0926 04:02:12.093223 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:12.093225 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:12.093319 (+ 94us) server_negotiation.cc:407] Connection header received
0926 04:02:12.093401 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:12.093402 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:12.093412 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:12.093504 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:12.094062 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:12.094536 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:12.095274 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:12.095466 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:12.095540 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:12.096701 (+ 1161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:12.096702 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:12.096703 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:02:12.096709 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:12.098101 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:12.098473 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:12.098474 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:12.098474 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:12.098530 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:12.098731 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:12.098732 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:12.098732 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:12.098873 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:12.098984 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:12.099088 (+ 104us) server_negotiation.cc:299] Negotiation successful
0926 04:02:12.099164 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 04:02:12.100977 17384 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60762:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 04:02:12.102540 17384 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 04:02:12.109365 17858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:12.104953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43280 (local address 127.0.98.2:44367)
0926 04:02:12.105608 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:12.105612 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:12.105632 (+ 20us) server_negotiation.cc:407] Connection header received
0926 04:02:12.105671 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:12.105673 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:12.105870 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:12.105958 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:02:12.106041 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:12.106716 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:12.108190 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:12.108997 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:12.109078 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:12.109137 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:12.109272 (+ 135us) server_negotiation.cc:299] Negotiation successful
0926 04:02:12.109307 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":79,"threads_started":1}
I20260926 04:02:12.109380 17856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:12.104799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44367 (local address 127.0.0.1:43280)
0926 04:02:12.104963 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:12.105002 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 04:02:12.105066 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:02:12.105952 (+ 886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:02:12.105952 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:02:12.105954 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:02:12.105989 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:12.105990 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:12.106828 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:12.106830 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:12.108085 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:12.108087 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:12.109085 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:12.109086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:12.109164 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:12.109179 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 04:02:12.109254 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 04:02:12.109303 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":55,"threads_started":1}
I20260926 04:02:12.109701 17857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:12.104828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45373 (local address 127.0.0.1:49308)
0926 04:02:12.105171 (+ 343us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:12.105179 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 04:02:12.105215 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:02:12.105755 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:02:12.105756 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:02:12.105758 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:02:12.105817 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:12.105819 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:12.107077 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:12.107078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:12.108183 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:12.108185 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:12.109490 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:12.109490 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:12.109548 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:12.109552 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 04:02:12.109603 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 04:02:12.109640 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"mutex_wait_us":12,"thread_start_us":126,"threads_started":1}
I20260926 04:02:12.109935 17860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:12.105365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49308 (local address 127.0.98.1:45373)
0926 04:02:12.105584 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:12.105589 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:12.105601 (+ 12us) server_negotiation.cc:407] Connection header received
0926 04:02:12.105645 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:12.105647 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:12.105664 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:12.105755 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:02:12.105881 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:12.106971 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:12.108271 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:12.108997 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:12.109078 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:12.109138 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:12.109797 (+ 659us) server_negotiation.cc:299] Negotiation successful
0926 04:02:12.109844 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":96,"threads_started":1}
I20260926 04:02:12.110235 17644 tablet_service.cc:1467] Processing CreateTablet for tablet 9ef7151d75434a6f8a658f642565acdd (DEFAULT_TABLE table=test-table [id=6e013d44d5274a4abc0d43d69c6f0e6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:02:12.110427 17855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:12.104828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43963 (local address 127.0.0.1:53266)
0926 04:02:12.105108 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:12.105117 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 04:02:12.105160 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:02:12.105672 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:02:12.105675 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:02:12.105678 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:02:12.105750 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:12.105752 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:12.108268 (+ 2516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:12.108269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:12.109017 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:12.109018 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:12.110218 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:12.110219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:12.110281 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:12.110285 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 04:02:12.110330 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 04:02:12.110366 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":87,"threads_started":1}
I20260926 04:02:12.110638 17644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ef7151d75434a6f8a658f642565acdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:02:12.110579 17510 tablet_service.cc:1467] Processing CreateTablet for tablet 9ef7151d75434a6f8a658f642565acdd (DEFAULT_TABLE table=test-table [id=6e013d44d5274a4abc0d43d69c6f0e6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:02:12.110725 17859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:12.105040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53266 (local address 127.0.98.3:43963)
0926 04:02:12.105314 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:12.105319 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:12.105331 (+ 12us) server_negotiation.cc:407] Connection header received
0926 04:02:12.105373 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:12.105376 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:12.105393 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:12.105509 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:02:12.105836 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:12.106971 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:12.109126 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:12.109853 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:12.109930 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:12.110564 (+ 634us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:12.110594 (+ 30us) server_negotiation.cc:299] Negotiation successful
0926 04:02:12.110634 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":201,"threads_started":1}
I20260926 04:02:12.110908 17510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ef7151d75434a6f8a658f642565acdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:02:12.111399 17778 tablet_service.cc:1467] Processing CreateTablet for tablet 9ef7151d75434a6f8a658f642565acdd (DEFAULT_TABLE table=test-table [id=6e013d44d5274a4abc0d43d69c6f0e6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:02:12.111712 17778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9ef7151d75434a6f8a658f642565acdd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:02:12.113467 17861 tablet_bootstrap.cc:492] T 9ef7151d75434a6f8a658f642565acdd P 0b85abd838e748acb7130d4327e4e6aa: Bootstrap starting.
I20260926 04:02:12.114044 17862 tablet_bootstrap.cc:492] T 9ef7151d75434a6f8a658f642565acdd P ddaa913c1bb44369a6d4f397848918b8: Bootstrap starting.
I20260926 04:02:12.114198 17861 tablet_bootstrap.cc:654] T 9ef7151d75434a6f8a658f642565acdd P 0b85abd838e748acb7130d4327e4e6aa: Neither blocks nor log segments found. Creating new log.
I20260926 04:02:12.114702 17862 tablet_bootstrap.cc:654] T 9ef7151d75434a6f8a658f642565acdd P ddaa913c1bb44369a6d4f397848918b8: Neither blocks nor log segments found. Creating new log.
I20260926 04:02:12.114749 17861 log.cc:824] T 9ef7151d75434a6f8a658f642565acdd P 0b85abd838e748acb7130d4327e4e6aa: Log is configured to *not* fsync() on all Append() calls
I20260926 04:02:12.115043 17863 tablet_bootstrap.cc:492] T 9ef7151d75434a6f8a658f642565acdd P 22ab4c101f2340978ee9395bad1f4d4c: Bootstrap starting.
I20260926 04:02:12.115319 17862 log.cc:824] T 9ef7151d75434a6f8a658f642565acdd P ddaa913c1bb44369a6d4f397848918b8: Log is configured to *not* fsync() on all Append() calls
I20260926 04:02:12.115617 17861 tablet_bootstrap.cc:492] T 9ef7151d75434a6f8a658f642565acdd P 0b85abd838e748acb7130d4327e4e6aa: No bootstrap required, opened a new log
I20260926 04:02:12.115712 17861 ts_tablet_manager.cc:1397] T 9ef7151d75434a6f8a658f642565acdd P 0b85abd838e748acb7130d4327e4e6aa: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 04:02:12.115747 17863 tablet_bootstrap.cc:654] T 9ef7151d75434a6f8a658f642565acdd P 22ab4c101f2340978ee9395bad1f4d4c: Neither blocks nor log segments found. Creating new log.
I20260926 04:02:12.116125 17862 tablet_bootstrap.cc:492] T 9ef7151d75434a6f8a658f642565acdd P ddaa913c1bb44369a6d4f397848918b8: No bootstrap required, opened a new log
I20260926 04:02:12.116201 17862 ts_tablet_manager.cc:1397] T 9ef7151d75434a6f8a658f642565acdd P ddaa913c1bb44369a6d4f397848918b8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 04:02:12.116226 17863 log.cc:824] T 9ef7151d75434a6f8a658f642565acdd P 22ab4c101f2340978ee9395bad1f4d4c: Log is configured to *not* fsync() on all Append() calls
I20260926 04:02:12.117111 17863 tablet_bootstrap.cc:492] T 9ef7151d75434a6f8a658f642565acdd P 22ab4c101f2340978ee9395bad1f4d4c: No bootstrap required, opened a new log
I20260926 04:02:12.117192 17863 ts_tablet_manager.cc:1397] T 9ef7151d75434a6f8a658f642565acdd P 22ab4c101f2340978ee9395bad1f4d4c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 04:02:12.117776 17861 raft_consensus.cc:348] T 9ef7151d75434a6f8a658f642565acdd P 0b85abd838e748acb7130d4327e4e6aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddaa913c1bb44369a6d4f397848918b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44367 } } peers { permanent_uuid: "0b85abd838e748acb7130d4327e4e6aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45373 } } peers { permanent_uuid: "22ab4c101f2340978ee9395bad1f4d4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43963 } }
I20260926 04:02:12.118016 17861 raft_consensus.cc:374] T 9ef7151d75434a6f8a658f642565acdd P 0b85abd838e748acb7130d4327e4e6aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:02:12.118053 17861 raft_consensus.cc:729] T 9ef7151d75434a6f8a658f642565acdd P 0b85abd838e748acb7130d4327e4e6aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b85abd838e748acb7130d4327e4e6aa, State: Initialized, Role: FOLLOWER
I20260926 04:02:12.118204 17861 consensus_queue.cc:260] T 9ef7151d75434a6f8a658f642565acdd P 0b85abd838e748acb7130d4327e4e6aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddaa913c1bb44369a6d4f397848918b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44367 } } peers { permanent_uuid: "0b85abd838e748acb7130d4327e4e6aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45373 } } peers { permanent_uuid: "22ab4c101f2340978ee9395bad1f4d4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43963 } }
I20260926 04:02:12.118203 17862 raft_consensus.cc:348] T 9ef7151d75434a6f8a658f642565acdd P ddaa913c1bb44369a6d4f397848918b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddaa913c1bb44369a6d4f397848918b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44367 } } peers { permanent_uuid: "0b85abd838e748acb7130d4327e4e6aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45373 } } peers { permanent_uuid: "22ab4c101f2340978ee9395bad1f4d4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43963 } }
I20260926 04:02:12.118384 17862 raft_consensus.cc:374] T 9ef7151d75434a6f8a658f642565acdd P ddaa913c1bb44369a6d4f397848918b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:02:12.118448 17862 raft_consensus.cc:729] T 9ef7151d75434a6f8a658f642565acdd P ddaa913c1bb44369a6d4f397848918b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddaa913c1bb44369a6d4f397848918b8, State: Initialized, Role: FOLLOWER
I20260926 04:02:12.118489 17861 ts_tablet_manager.cc:1428] T 9ef7151d75434a6f8a658f642565acdd P 0b85abd838e748acb7130d4327e4e6aa: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 04:02:12.118556 17862 consensus_queue.cc:260] T 9ef7151d75434a6f8a658f642565acdd P ddaa913c1bb44369a6d4f397848918b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddaa913c1bb44369a6d4f397848918b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44367 } } peers { permanent_uuid: "0b85abd838e748acb7130d4327e4e6aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45373 } } peers { permanent_uuid: "22ab4c101f2340978ee9395bad1f4d4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43963 } }
I20260926 04:02:12.118616 17576 heartbeater.cc:503] Master 127.0.98.62:35803 was elected leader, sending a full tablet report...
I20260926 04:02:12.118813 17862 ts_tablet_manager.cc:1428] T 9ef7151d75434a6f8a658f642565acdd P ddaa913c1bb44369a6d4f397848918b8: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 04:02:12.118819 17710 heartbeater.cc:503] Master 127.0.98.62:35803 was elected leader, sending a full tablet report...
I20260926 04:02:12.119168 17863 raft_consensus.cc:348] T 9ef7151d75434a6f8a658f642565acdd P 22ab4c101f2340978ee9395bad1f4d4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddaa913c1bb44369a6d4f397848918b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44367 } } peers { permanent_uuid: "0b85abd838e748acb7130d4327e4e6aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45373 } } peers { permanent_uuid: "22ab4c101f2340978ee9395bad1f4d4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43963 } }
I20260926 04:02:12.119376 17863 raft_consensus.cc:374] T 9ef7151d75434a6f8a658f642565acdd P 22ab4c101f2340978ee9395bad1f4d4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:02:12.119434 17863 raft_consensus.cc:729] T 9ef7151d75434a6f8a658f642565acdd P 22ab4c101f2340978ee9395bad1f4d4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22ab4c101f2340978ee9395bad1f4d4c, State: Initialized, Role: FOLLOWER
I20260926 04:02:12.119666 17863 consensus_queue.cc:260] T 9ef7151d75434a6f8a658f642565acdd P 22ab4c101f2340978ee9395bad1f4d4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddaa913c1bb44369a6d4f397848918b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44367 } } peers { permanent_uuid: "0b85abd838e748acb7130d4327e4e6aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45373 } } peers { permanent_uuid: "22ab4c101f2340978ee9395bad1f4d4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43963 } }
I20260926 04:02:12.119946 17844 heartbeater.cc:503] Master 127.0.98.62:35803 was elected leader, sending a full tablet report...
I20260926 04:02:12.119971 17863 ts_tablet_manager.cc:1428] T 9ef7151d75434a6f8a658f642565acdd P 22ab4c101f2340978ee9395bad1f4d4c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20260926 04:02:12.152527 17711 tablet.cc:2369] T 9ef7151d75434a6f8a658f642565acdd P ddaa913c1bb44369a6d4f397848918b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 04:02:12.247872 17577 tablet.cc:2369] T 9ef7151d75434a6f8a658f642565acdd P 0b85abd838e748acb7130d4327e4e6aa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 04:02:12.263176 17868 raft_consensus.cc:482] T 9ef7151d75434a6f8a658f642565acdd P ddaa913c1bb44369a6d4f397848918b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 04:02:12.263298 17868 raft_consensus.cc:504] T 9ef7151d75434a6f8a658f642565acdd P ddaa913c1bb44369a6d4f397848918b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddaa913c1bb44369a6d4f397848918b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44367 } } peers { permanent_uuid: "0b85abd838e748acb7130d4327e4e6aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45373 } } peers { permanent_uuid: "22ab4c101f2340978ee9395bad1f4d4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43963 } }
I20260926 04:02:12.263720 17868 leader_election.cc:288] T 9ef7151d75434a6f8a658f642565acdd P ddaa913c1bb44369a6d4f397848918b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0b85abd838e748acb7130d4327e4e6aa (127.0.98.1:45373), 22ab4c101f2340978ee9395bad1f4d4c (127.0.98.3:43963)
I20260926 04:02:12.267546 17859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:12.264048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46231 (local address 127.0.98.3:43963)
0926 04:02:12.264129 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:12.264132 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:12.264141 (+ 9us) server_negotiation.cc:407] Connection header received
0926 04:02:12.264225 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:12.264226 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:12.264242 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:12.264284 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:02:12.264439 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:12.265148 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:12.266626 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:12.267174 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:12.267252 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:12.267261 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:12.267430 (+ 169us) server_negotiation.cc:299] Negotiation successful
0926 04:02:12.267466 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 04:02:12.267539 17860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:12.263954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59851 (local address 127.0.98.1:45373)
0926 04:02:12.263996 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:12.263998 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:12.264055 (+ 57us) server_negotiation.cc:407] Connection header received
0926 04:02:12.264089 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:12.264090 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:12.264103 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:12.264160 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:02:12.264366 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:12.265149 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:12.266627 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:12.267176 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:12.267252 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:12.267261 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:12.267430 (+ 169us) server_negotiation.cc:299] Negotiation successful
0926 04:02:12.267466 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 04:02:12.267603 17712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:12.263888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45373 (local address 127.0.98.2:59851)
0926 04:02:12.263945 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:12.263958 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 04:02:12.264004 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:02:12.264185 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:02:12.264186 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:02:12.264189 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:02:12.264263 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:12.264265 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:12.265234 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:12.265236 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:12.266528 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:12.266530 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:12.267261 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:12.267262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:12.267347 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:12.267352 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 04:02:12.267411 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 04:02:12.267460 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":48}
I20260926 04:02:12.267609 17870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:12.263898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43963 (local address 127.0.98.2:46231)
0926 04:02:12.264069 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:12.264079 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 04:02:12.264155 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:02:12.264306 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:02:12.264307 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:02:12.264308 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:02:12.264368 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:12.264369 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:12.265234 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:12.265236 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:12.266528 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:12.266530 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:12.267262 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:12.267262 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:12.267347 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:12.267352 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 04:02:12.267411 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 04:02:12.267462 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":61,"thread_start_us":99,"threads_started":1}
I20260926 04:02:12.267967 17530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ef7151d75434a6f8a658f642565acdd" candidate_uuid: "ddaa913c1bb44369a6d4f397848918b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b85abd838e748acb7130d4327e4e6aa" is_pre_election: true
I20260926 04:02:12.268136 17530 raft_consensus.cc:2454] T 9ef7151d75434a6f8a658f642565acdd P 0b85abd838e748acb7130d4327e4e6aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddaa913c1bb44369a6d4f397848918b8 in term 0.
I20260926 04:02:12.268111 17798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ef7151d75434a6f8a658f642565acdd" candidate_uuid: "ddaa913c1bb44369a6d4f397848918b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22ab4c101f2340978ee9395bad1f4d4c" is_pre_election: true
I20260926 04:02:12.268261 17798 raft_consensus.cc:2454] T 9ef7151d75434a6f8a658f642565acdd P 22ab4c101f2340978ee9395bad1f4d4c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ddaa913c1bb44369a6d4f397848918b8 in term 0.
I20260926 04:02:12.268394 17600 leader_election.cc:302] T 9ef7151d75434a6f8a658f642565acdd P ddaa913c1bb44369a6d4f397848918b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b85abd838e748acb7130d4327e4e6aa, ddaa913c1bb44369a6d4f397848918b8; no voters:
I20260926 04:02:12.268544 17868 raft_consensus.cc:2781] T 9ef7151d75434a6f8a658f642565acdd P ddaa913c1bb44369a6d4f397848918b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 04:02:12.268630 17868 raft_consensus.cc:482] T 9ef7151d75434a6f8a658f642565acdd P ddaa913c1bb44369a6d4f397848918b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 04:02:12.268687 17868 raft_consensus.cc:3037] T 9ef7151d75434a6f8a658f642565acdd P ddaa913c1bb44369a6d4f397848918b8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:02:12.269378 17868 raft_consensus.cc:504] T 9ef7151d75434a6f8a658f642565acdd P ddaa913c1bb44369a6d4f397848918b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddaa913c1bb44369a6d4f397848918b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44367 } } peers { permanent_uuid: "0b85abd838e748acb7130d4327e4e6aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45373 } } peers { permanent_uuid: "22ab4c101f2340978ee9395bad1f4d4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43963 } }
I20260926 04:02:12.269534 17868 leader_election.cc:288] T 9ef7151d75434a6f8a658f642565acdd P ddaa913c1bb44369a6d4f397848918b8 [CANDIDATE]: Term 1 election: Requested vote from peers 0b85abd838e748acb7130d4327e4e6aa (127.0.98.1:45373), 22ab4c101f2340978ee9395bad1f4d4c (127.0.98.3:43963)
I20260926 04:02:12.269707 17798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ef7151d75434a6f8a658f642565acdd" candidate_uuid: "ddaa913c1bb44369a6d4f397848918b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22ab4c101f2340978ee9395bad1f4d4c"
I20260926 04:02:12.269726 17530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9ef7151d75434a6f8a658f642565acdd" candidate_uuid: "ddaa913c1bb44369a6d4f397848918b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0b85abd838e748acb7130d4327e4e6aa"
I20260926 04:02:12.269799 17798 raft_consensus.cc:3037] T 9ef7151d75434a6f8a658f642565acdd P 22ab4c101f2340978ee9395bad1f4d4c [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:02:12.269794 17530 raft_consensus.cc:3037] T 9ef7151d75434a6f8a658f642565acdd P 0b85abd838e748acb7130d4327e4e6aa [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:02:12.270521 17798 raft_consensus.cc:2454] T 9ef7151d75434a6f8a658f642565acdd P 22ab4c101f2340978ee9395bad1f4d4c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddaa913c1bb44369a6d4f397848918b8 in term 1.
I20260926 04:02:12.270524 17530 raft_consensus.cc:2454] T 9ef7151d75434a6f8a658f642565acdd P 0b85abd838e748acb7130d4327e4e6aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ddaa913c1bb44369a6d4f397848918b8 in term 1.
I20260926 04:02:12.270690 17600 leader_election.cc:302] T 9ef7151d75434a6f8a658f642565acdd P ddaa913c1bb44369a6d4f397848918b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b85abd838e748acb7130d4327e4e6aa, ddaa913c1bb44369a6d4f397848918b8; no voters:
I20260926 04:02:12.270810 17868 raft_consensus.cc:2781] T 9ef7151d75434a6f8a658f642565acdd P ddaa913c1bb44369a6d4f397848918b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:02:12.270946 17868 raft_consensus.cc:686] T 9ef7151d75434a6f8a658f642565acdd P ddaa913c1bb44369a6d4f397848918b8 [term 1 LEADER]: Becoming Leader. State: Replica: ddaa913c1bb44369a6d4f397848918b8, State: Running, Role: LEADER
I20260926 04:02:12.271049 17868 consensus_queue.cc:237] T 9ef7151d75434a6f8a658f642565acdd P ddaa913c1bb44369a6d4f397848918b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddaa913c1bb44369a6d4f397848918b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44367 } } peers { permanent_uuid: "0b85abd838e748acb7130d4327e4e6aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45373 } } peers { permanent_uuid: "22ab4c101f2340978ee9395bad1f4d4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43963 } }
I20260926 04:02:12.271858 17383 catalog_manager.cc:5495] T 9ef7151d75434a6f8a658f642565acdd P ddaa913c1bb44369a6d4f397848918b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to ddaa913c1bb44369a6d4f397848918b8 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "ddaa913c1bb44369a6d4f397848918b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ddaa913c1bb44369a6d4f397848918b8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44367 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b85abd838e748acb7130d4327e4e6aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "22ab4c101f2340978ee9395bad1f4d4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43963 } health_report { overall_health: UNKNOWN } } }
I20260926 04:02:12.293751 17858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:12.291046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43292 (local address 127.0.98.2:44367)
0926 04:02:12.291111 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:12.291114 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:12.291123 (+ 9us) server_negotiation.cc:407] Connection header received
0926 04:02:12.291199 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:12.291200 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:12.291221 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:12.291309 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 04:02:12.291439 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:12.291910 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:12.292853 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:12.293071 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:12.293156 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:12.293290 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 04:02:12.293474 (+ 184us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 04:02:12.293588 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:12.293611 (+ 23us) server_negotiation.cc:299] Negotiation successful
0926 04:02:12.293632 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 04:02:12.295542 17798 raft_consensus.cc:1261] T 9ef7151d75434a6f8a658f642565acdd P 22ab4c101f2340978ee9395bad1f4d4c [term 1 FOLLOWER]: Refusing update from remote peer ddaa913c1bb44369a6d4f397848918b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:02:12.295573 17530 raft_consensus.cc:1261] T 9ef7151d75434a6f8a658f642565acdd P 0b85abd838e748acb7130d4327e4e6aa [term 1 FOLLOWER]: Refusing update from remote peer ddaa913c1bb44369a6d4f397848918b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:02:12.295912 17868 consensus_queue.cc:1035] T 9ef7151d75434a6f8a658f642565acdd P ddaa913c1bb44369a6d4f397848918b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "22ab4c101f2340978ee9395bad1f4d4c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:02:12.296083 17871 consensus_queue.cc:1035] T 9ef7151d75434a6f8a658f642565acdd P ddaa913c1bb44369a6d4f397848918b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0b85abd838e748acb7130d4327e4e6aa" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:02:12.299290 17876 mvcc.cc:205] Tried to move back new op lower bound from 7333459281079894016 to 7333459280982814720. Current Snapshot: MvccSnapshot[applied={T|T < 7333459281079894016}]
I20260926 04:02:12.298875 17875 mvcc.cc:205] Tried to move back new op lower bound from 7333459281079894016 to 7333459280982814720. Current Snapshot: MvccSnapshot[applied={T|T < 7333459281079894016}]
I20260926 04:02:12.300470 17383 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 04:02:12.300565 17383 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 04:02:12.301294 17383 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2f1181ce31be4563a3ef730c7a5fa4f9: Sending DeleteTablet for 3 replicas of tablet 9ef7151d75434a6f8a658f642565acdd
I20260926 04:02:12.301616 17778 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ef7151d75434a6f8a658f642565acdd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:02:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53266
I20260926 04:02:12.301632 17644 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ef7151d75434a6f8a658f642565acdd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:02:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43280
I20260926 04:02:12.301864 17884 tablet_replica.cc:330] T 9ef7151d75434a6f8a658f642565acdd P ddaa913c1bb44369a6d4f397848918b8: stopping tablet replica
I20260926 04:02:12.301896 17510 tablet_service.cc:1514] Processing DeleteTablet for tablet 9ef7151d75434a6f8a658f642565acdd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:02:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49308
I20260926 04:02:12.301963 17884 raft_consensus.cc:2229] T 9ef7151d75434a6f8a658f642565acdd P ddaa913c1bb44369a6d4f397848918b8 [term 1 LEADER]: Raft consensus shutting down.
I20260926 04:02:12.302048 17884 raft_consensus.cc:2258] T 9ef7151d75434a6f8a658f642565acdd P ddaa913c1bb44369a6d4f397848918b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:02:12.302175 17886 tablet_replica.cc:330] T 9ef7151d75434a6f8a658f642565acdd P 0b85abd838e748acb7130d4327e4e6aa: stopping tablet replica
I20260926 04:02:12.302179 17885 tablet_replica.cc:330] T 9ef7151d75434a6f8a658f642565acdd P 22ab4c101f2340978ee9395bad1f4d4c: stopping tablet replica
I20260926 04:02:12.302264 17885 raft_consensus.cc:2229] T 9ef7151d75434a6f8a658f642565acdd P 22ab4c101f2340978ee9395bad1f4d4c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 04:02:12.302312 17886 raft_consensus.cc:2229] T 9ef7151d75434a6f8a658f642565acdd P 0b85abd838e748acb7130d4327e4e6aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 04:02:12.302389 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 17447
I20260926 04:02:12.302421 17885 raft_consensus.cc:2258] T 9ef7151d75434a6f8a658f642565acdd P 22ab4c101f2340978ee9395bad1f4d4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:02:12.303742 17885 ts_tablet_manager.cc:1905] T 9ef7151d75434a6f8a658f642565acdd P 22ab4c101f2340978ee9395bad1f4d4c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:02:12.304247 17884 ts_tablet_manager.cc:1905] T 9ef7151d75434a6f8a658f642565acdd P ddaa913c1bb44369a6d4f397848918b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:02:12.305428 17885 ts_tablet_manager.cc:1918] T 9ef7151d75434a6f8a658f642565acdd P 22ab4c101f2340978ee9395bad1f4d4c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 04:02:12.305541 17885 log.cc:1196] T 9ef7151d75434a6f8a658f642565acdd P 22ab4c101f2340978ee9395bad1f4d4c: Deleting WAL directory at /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/minicluster-data/ts-2/wal/wals/9ef7151d75434a6f8a658f642565acdd
I20260926 04:02:12.305857 17885 ts_tablet_manager.cc:1939] T 9ef7151d75434a6f8a658f642565acdd P 22ab4c101f2340978ee9395bad1f4d4c: Deleting consensus metadata
I20260926 04:02:12.306069 17884 ts_tablet_manager.cc:1918] T 9ef7151d75434a6f8a658f642565acdd P ddaa913c1bb44369a6d4f397848918b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 04:02:12.306154 17884 log.cc:1196] T 9ef7151d75434a6f8a658f642565acdd P ddaa913c1bb44369a6d4f397848918b8: Deleting WAL directory at /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790395095474643-392-0/minicluster-data/ts-1/wal/wals/9ef7151d75434a6f8a658f642565acdd
I20260926 04:02:12.306288 17369 catalog_manager.cc:4841] TS 22ab4c101f2340978ee9395bad1f4d4c (127.0.98.3:43963): tablet 9ef7151d75434a6f8a658f642565acdd (table test-table [id=6e013d44d5274a4abc0d43d69c6f0e6d]) successfully deleted
I20260926 04:02:12.306435 17884 ts_tablet_manager.cc:1939] T 9ef7151d75434a6f8a658f642565acdd P ddaa913c1bb44369a6d4f397848918b8: Deleting consensus metadata
I20260926 04:02:12.306816 17371 catalog_manager.cc:4841] TS ddaa913c1bb44369a6d4f397848918b8 (127.0.98.2:44367): tablet 9ef7151d75434a6f8a658f642565acdd (table test-table [id=6e013d44d5274a4abc0d43d69c6f0e6d]) successfully deleted
W20260926 04:02:12.309803 17372 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45373 (error 108)
I20260926 04:02:12.309998 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 17581
I20260926 04:02:12.311005 17855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:12.310862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45373 (local address 127.0.0.1:49314)
0926 04:02:12.310926 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:12.310972 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45373: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 04:02:12.311143 17372 catalog_manager.cc:4584] TS 0b85abd838e748acb7130d4327e4e6aa (127.0.98.1:45373): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9ef7151d75434a6f8a658f642565acdd: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45373: connect: Connection refused (error 111)
I20260926 04:02:12.314558 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 17715
I20260926 04:02:12.319135 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 17353
2026-09-26T04:02:12Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2015 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-26T04:02:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:02:12Z Disabled control of system clock
I20260926 04:02:12.345918 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395095474643-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395095474643-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395095474643-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33715
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33715
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 04:02:12.426491 17891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:02:12.426702 17891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:02:12.426725 17891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:02:12.427796 17891 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:02:12.427830 17891 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:02:12.427843 17891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:02:12.427856 17891 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:02:12.427868 17891 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:02:12.429134 17891 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395095474643-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395095474643-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33715
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33715
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17891
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395095474643-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:02:12.429373 17891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:02:12.429642 17891 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:02:12.434379 17897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:02:12.434383 17896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:02:12.434398 17899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:02:12.434639 17891 server_base.cc:1034] running on GCE node
I20260926 04:02:12.434917 17891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:02:12.435207 17891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:02:12.436350 17891 hybrid_clock.cc:648] HybridClock initialized: now 1790395332436343 us; error 31 us; skew 500 ppm
I20260926 04:02:12.437572 17891 webserver.cc:460] Webserver started at http://127.0.98.62:35297/ using document root <none> and password file <none>
I20260926 04:02:12.437846 17891 fs_manager.cc:359] Metadata directory not provided
I20260926 04:02:12.437892 17891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:02:12.438081 17891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:02:12.439114 17891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395095474643-392-0/minicluster-data/master-0/data/instance:
uuid: "b9b760b328814479ba55d27e0d56da67"
format_stamp: "Formatted at 2026-09-26 04:02:12 on dist-test-slave-nhpw"
server_key: "83ba3284e6cc84ebeb09a701f8e1bc6d"
server_key_iv: "7275e287d37f0a7cee3ae9b40f713124"
server_key_version: "encryptionkey@0"
I20260926 04:02:12.439481 17891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395095474643-392-0/minicluster-data/master-0/wal/instance:
uuid: "b9b760b328814479ba55d27e0d56da67"
format_stamp: "Formatted at 2026-09-26 04:02:12 on dist-test-slave-nhpw"
server_key: "83ba3284e6cc84ebeb09a701f8e1bc6d"
server_key_iv: "7275e287d37f0a7cee3ae9b40f713124"
server_key_version: "encryptionkey@0"
I20260926 04:02:12.440789 17891 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 04:02:12.441525 17905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:12.441748 17891 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:02:12.441854 17891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395095474643-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395095474643-392-0/minicluster-data/master-0/wal
uuid: "b9b760b328814479ba55d27e0d56da67"
format_stamp: "Formatted at 2026-09-26 04:02:12 on dist-test-slave-nhpw"
server_key: "83ba3284e6cc84ebeb09a701f8e1bc6d"
server_key_iv: "7275e287d37f0a7cee3ae9b40f713124"
server_key_version: "encryptionkey@0"
I20260926 04:02:12.441926 17891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395095474643-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395095474643-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395095474643-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:02:12.451556 17891 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:02:12.451866 17891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:02:12.452037 17891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:02:12.456068 17891 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33715
I20260926 04:02:12.456100 17957 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33715 every 8 connection(s)
I20260926 04:02:12.456429 17891 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
I20260926 04:02:12.457420 17958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:02:12.459583 17958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9b760b328814479ba55d27e0d56da67: Bootstrap starting.
I20260926 04:02:12.459892 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 17891
I20260926 04:02:12.459996 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395095474643-392-0/minicluster-data/master-0/wal/instance
I20260926 04:02:12.460170 17958 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9b760b328814479ba55d27e0d56da67: Neither blocks nor log segments found. Creating new log.
I20260926 04:02:12.460175 392 external_mini_cluster.cc:1428] Setting key a99018aecce6aec1c1238d2bd2cb9647
I20260926 04:02:12.460474 17958 log.cc:824] T 00000000000000000000000000000000 P b9b760b328814479ba55d27e0d56da67: Log is configured to *not* fsync() on all Append() calls
I20260926 04:02:12.461190 17958 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9b760b328814479ba55d27e0d56da67: No bootstrap required, opened a new log
I20260926 04:02:12.462915 17958 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9b760b328814479ba55d27e0d56da67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9b760b328814479ba55d27e0d56da67" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33715 } }
I20260926 04:02:12.463075 17958 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9b760b328814479ba55d27e0d56da67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:02:12.463124 17958 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9b760b328814479ba55d27e0d56da67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9b760b328814479ba55d27e0d56da67, State: Initialized, Role: FOLLOWER
I20260926 04:02:12.463263 17958 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9b760b328814479ba55d27e0d56da67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9b760b328814479ba55d27e0d56da67" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33715 } }
I20260926 04:02:12.463341 17958 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9b760b328814479ba55d27e0d56da67 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:02:12.463384 17958 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9b760b328814479ba55d27e0d56da67 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:02:12.463430 17958 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9b760b328814479ba55d27e0d56da67 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:02:12.463951 17958 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9b760b328814479ba55d27e0d56da67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9b760b328814479ba55d27e0d56da67" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33715 } }
I20260926 04:02:12.464071 17958 leader_election.cc:302] T 00000000000000000000000000000000 P b9b760b328814479ba55d27e0d56da67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9b760b328814479ba55d27e0d56da67; no voters:
I20260926 04:02:12.464272 17958 leader_election.cc:288] T 00000000000000000000000000000000 P b9b760b328814479ba55d27e0d56da67 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 04:02:12.464298 17963 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9b760b328814479ba55d27e0d56da67 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:02:12.464479 17963 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9b760b328814479ba55d27e0d56da67 [term 1 LEADER]: Becoming Leader. State: Replica: b9b760b328814479ba55d27e0d56da67, State: Running, Role: LEADER
I20260926 04:02:12.464602 17958 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9b760b328814479ba55d27e0d56da67 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:02:12.464936 17963 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9b760b328814479ba55d27e0d56da67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9b760b328814479ba55d27e0d56da67" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33715 } }
I20260926 04:02:12.465411 17965 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9b760b328814479ba55d27e0d56da67 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9b760b328814479ba55d27e0d56da67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9b760b328814479ba55d27e0d56da67" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33715 } } }
I20260926 04:02:12.465487 17965 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9b760b328814479ba55d27e0d56da67 [sys.catalog]: This master's current role is: LEADER
I20260926 04:02:12.465627 17964 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9b760b328814479ba55d27e0d56da67 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9b760b328814479ba55d27e0d56da67. Latest consensus state: current_term: 1 leader_uuid: "b9b760b328814479ba55d27e0d56da67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9b760b328814479ba55d27e0d56da67" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33715 } } }
I20260926 04:02:12.465687 17964 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9b760b328814479ba55d27e0d56da67 [sys.catalog]: This master's current role is: LEADER
I20260926 04:02:12.465999 17962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:12.460655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44092 (local address 127.0.98.62:33715)
0926 04:02:12.460843 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:12.460853 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:12.460888 (+ 35us) server_negotiation.cc:407] Connection header received
0926 04:02:12.461359 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:12.461366 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:12.461427 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:12.461572 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:12.461828 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:12.462690 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:12.463725 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:12.464198 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:12.464345 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:12.465093 (+ 748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:12.465096 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:12.465096 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:02:12.465314 (+ 218us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:12.465359 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:02:12.465375 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:12.465683 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:12.465715 (+ 32us) server_negotiation.cc:299] Negotiation successful
0926 04:02:12.465797 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":67,"threads_started":1}
I20260926 04:02:12.466071 17972 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:02:12.466506 17972 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:02:12.468147 17972 catalog_manager.cc:1347] Generated new cluster ID: d3e7492d796545c4a9e38aa16db7d517
I20260926 04:02:12.468191 17972 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 04:02:12.492241 17972 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:02:12.492863 17972 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:02:12.496779 17972 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b9b760b328814479ba55d27e0d56da67: Generated new TSK 0
I20260926 04:02:12.496965 17972 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 04:02:12.502885 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395095474643-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395095474643-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395095474643-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33715
--builtin_ntp_servers=127.0.98.20:35107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 04:02:12.582365 17982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:02:12.582579 17982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:02:12.582602 17982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:02:12.583758 17982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:02:12.583799 17982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:02:12.583825 17982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 04:02:12.585218 17982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395095474643-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395095474643-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395095474643-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:02:12.585459 17982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:02:12.585757 17982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:02:12.590750 17988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:02:12.590826 17982 server_base.cc:1034] running on GCE node
W20260926 04:02:12.590729 17987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:02:12.590718 17990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:02:12.591141 17982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:02:12.591379 17982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:02:12.592518 17982 hybrid_clock.cc:648] HybridClock initialized: now 1790395332592502 us; error 32 us; skew 500 ppm
I20260926 04:02:12.593813 17982 webserver.cc:460] Webserver started at http://127.0.98.1:33731/ using document root <none> and password file <none>
I20260926 04:02:12.594048 17982 fs_manager.cc:359] Metadata directory not provided
I20260926 04:02:12.594097 17982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:02:12.594269 17982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:02:12.595300 17982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395095474643-392-0/minicluster-data/ts-0/data/instance:
uuid: "13c199659db54660ad5cf0c3151f7046"
format_stamp: "Formatted at 2026-09-26 04:02:12 on dist-test-slave-nhpw"
server_key: "30e95d903c13c586859c17bc9498c128"
server_key_iv: "ffd651a1f0db59ccee1eed6280f389d9"
server_key_version: "encryptionkey@0"
I20260926 04:02:12.595656 17982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395095474643-392-0/minicluster-data/ts-0/wal/instance:
uuid: "13c199659db54660ad5cf0c3151f7046"
format_stamp: "Formatted at 2026-09-26 04:02:12 on dist-test-slave-nhpw"
server_key: "30e95d903c13c586859c17bc9498c128"
server_key_iv: "ffd651a1f0db59ccee1eed6280f389d9"
server_key_version: "encryptionkey@0"
I20260926 04:02:12.597016 17982 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 04:02:12.597728 17996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:12.597913 17982 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 04:02:12.598002 17982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395095474643-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395095474643-392-0/minicluster-data/ts-0/wal
uuid: "13c199659db54660ad5cf0c3151f7046"
format_stamp: "Formatted at 2026-09-26 04:02:12 on dist-test-slave-nhpw"
server_key: "30e95d903c13c586859c17bc9498c128"
server_key_iv: "ffd651a1f0db59ccee1eed6280f389d9"
server_key_version: "encryptionkey@0"
I20260926 04:02:12.598074 17982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395095474643-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395095474643-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395095474643-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:02:12.617755 17982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:02:12.618090 17982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:02:12.618247 17982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:02:12.618480 17982 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:02:12.618800 17982 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:02:12.618834 17982 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:12.618877 17982 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:02:12.618904 17982 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:12.625773 17982 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45905
I20260926 04:02:12.625862 18109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45905 every 8 connection(s)
I20260926 04:02:12.626266 17982 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
I20260926 04:02:12.627666 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 17982
I20260926 04:02:12.627759 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395095474643-392-0/minicluster-data/ts-0/wal/instance
I20260926 04:02:12.627967 392 external_mini_cluster.cc:1428] Setting key 1ac377ba1639efacafb63d96beb2eb02
I20260926 04:02:12.629474 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395095474643-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395095474643-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395095474643-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33715
--builtin_ntp_servers=127.0.98.20:35107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 04:02:12.632998 17962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:12.627298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38467 (local address 127.0.98.62:33715)
0926 04:02:12.627371 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:12.627373 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:12.627385 (+ 12us) server_negotiation.cc:407] Connection header received
0926 04:02:12.628028 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:12.628030 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:12.628042 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:12.628115 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:12.629043 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:12.629772 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:12.631281 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:12.631658 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:12.631705 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:12.632764 (+ 1059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:12.632766 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:12.632766 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:02:12.632770 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:12.632784 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:02:12.632788 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:12.632829 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:12.632921 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 04:02:12.632941 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 04:02:12.633287 18112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:12.627011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33715 (local address 127.0.98.1:38467)
0926 04:02:12.627226 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:12.627272 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 04:02:12.627901 (+ 629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:02:12.628191 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:02:12.628196 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:02:12.628206 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:02:12.628944 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:12.628950 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:12.629879 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:12.629880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:12.631168 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:12.631172 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:12.631802 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:12.631803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:12.631958 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:12.632572 (+ 614us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 04:02:12.632573 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:02:12.632611 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 04:02:12.632667 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:02:12.632848 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:02:12.632851 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:02:12.632854 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 04:02:12.632925 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 04:02:12.633032 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":87,"threads_started":1}
I20260926 04:02:12.633785 18110 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33715
I20260926 04:02:12.633909 18110 heartbeater.cc:464] Registering TS with master...
I20260926 04:02:12.634083 18110 heartbeater.cc:511] Master 127.0.98.62:33715 requested a full tablet report, sending...
I20260926 04:02:12.634670 17922 ts_manager.cc:194] Registered new tserver with Master: 13c199659db54660ad5cf0c3151f7046 (127.0.98.1:45905)
I20260926 04:02:12.635671 17922 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:38467
W20260926 04:02:12.711047 18113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:02:12.711239 18113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:02:12.711259 18113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:02:12.712303 18113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:02:12.712334 18113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:02:12.712357 18113 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 04:02:12.713610 18113 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395095474643-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395095474643-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18113
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395095474643-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:02:12.713779 18113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:02:12.714030 18113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:02:12.718791 18118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:02:12.718780 18121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:02:12.718883 18113 server_base.cc:1034] running on GCE node
W20260926 04:02:12.718776 18119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:02:12.719290 18113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:02:12.719522 18113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:02:12.720700 18113 hybrid_clock.cc:648] HybridClock initialized: now 1790395332720694 us; error 35 us; skew 500 ppm
I20260926 04:02:12.721982 18113 webserver.cc:460] Webserver started at http://127.0.98.2:43523/ using document root <none> and password file <none>
I20260926 04:02:12.722221 18113 fs_manager.cc:359] Metadata directory not provided
I20260926 04:02:12.722290 18113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:02:12.722429 18113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:02:12.723491 18113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395095474643-392-0/minicluster-data/ts-1/data/instance:
uuid: "aed2e00ab7e94d2786e20459857979c0"
format_stamp: "Formatted at 2026-09-26 04:02:12 on dist-test-slave-nhpw"
server_key: "d94bf04fb4a3e0c3649797619d436d7f"
server_key_iv: "bc0db075bf0da64d2b0712fdc98ab131"
server_key_version: "encryptionkey@0"
I20260926 04:02:12.723846 18113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395095474643-392-0/minicluster-data/ts-1/wal/instance:
uuid: "aed2e00ab7e94d2786e20459857979c0"
format_stamp: "Formatted at 2026-09-26 04:02:12 on dist-test-slave-nhpw"
server_key: "d94bf04fb4a3e0c3649797619d436d7f"
server_key_iv: "bc0db075bf0da64d2b0712fdc98ab131"
server_key_version: "encryptionkey@0"
I20260926 04:02:12.725229 18113 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 04:02:12.725927 18127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:12.726082 18113 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 04:02:12.726176 18113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395095474643-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395095474643-392-0/minicluster-data/ts-1/wal
uuid: "aed2e00ab7e94d2786e20459857979c0"
format_stamp: "Formatted at 2026-09-26 04:02:12 on dist-test-slave-nhpw"
server_key: "d94bf04fb4a3e0c3649797619d436d7f"
server_key_iv: "bc0db075bf0da64d2b0712fdc98ab131"
server_key_version: "encryptionkey@0"
I20260926 04:02:12.726230 18113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395095474643-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395095474643-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395095474643-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:02:12.739933 18113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:02:12.740160 18113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:02:12.740255 18113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:02:12.740418 18113 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:02:12.740815 18113 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:02:12.740854 18113 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:12.740902 18113 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:02:12.740924 18113 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:12.746946 18113 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35187
I20260926 04:02:12.746997 18240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35187 every 8 connection(s)
I20260926 04:02:12.747304 18113 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
I20260926 04:02:12.752141 17962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:12.747988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38457 (local address 127.0.98.62:33715)
0926 04:02:12.748035 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:12.748037 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:12.748265 (+ 228us) server_negotiation.cc:407] Connection header received
0926 04:02:12.748785 (+ 520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:12.748786 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:12.748797 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:12.748852 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:12.749473 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:12.749953 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:12.750870 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:12.751057 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:12.751107 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:12.751913 (+ 806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:12.751914 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:12.751915 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:02:12.751918 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:12.751929 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:02:12.751932 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:12.751970 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:12.752065 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 04:02:12.752083 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 04:02:12.752317 18243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:12.747953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33715 (local address 127.0.98.2:38457)
0926 04:02:12.748134 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:12.748172 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 04:02:12.748701 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:02:12.748910 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:02:12.748914 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:02:12.748922 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:02:12.749402 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:12.749406 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:12.750043 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:12.750044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:12.750793 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:12.750795 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:12.751130 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:12.751131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:12.751234 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:12.751787 (+ 553us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 04:02:12.751788 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:02:12.751815 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 04:02:12.751835 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:02:12.751996 (+ 161us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:02:12.751998 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:02:12.752001 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 04:02:12.752049 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 04:02:12.752121 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":93,"threads_started":1}
I20260926 04:02:12.752897 18241 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33715
I20260926 04:02:12.753015 18241 heartbeater.cc:464] Registering TS with master...
I20260926 04:02:12.753196 18241 heartbeater.cc:511] Master 127.0.98.62:33715 requested a full tablet report, sending...
I20260926 04:02:12.753504 17922 ts_manager.cc:194] Registered new tserver with Master: aed2e00ab7e94d2786e20459857979c0 (127.0.98.2:35187)
I20260926 04:02:12.753831 17922 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:38457
I20260926 04:02:12.755175 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 18113
I20260926 04:02:12.755282 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395095474643-392-0/minicluster-data/ts-1/wal/instance
I20260926 04:02:12.755462 392 external_mini_cluster.cc:1428] Setting key f361da659e89cae94ebdbd4bb7694755
I20260926 04:02:12.756309 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395095474643-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395095474643-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395095474643-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33715
--builtin_ntp_servers=127.0.98.20:35107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 04:02:12.836306 18244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:02:12.836500 18244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:02:12.836521 18244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:02:12.837668 18244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:02:12.837709 18244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:02:12.837733 18244 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 04:02:12.839071 18244 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395095474643-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395095474643-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33715
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18244
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395095474643-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:02:12.839308 18244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:02:12.839592 18244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:02:12.844471 18250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:02:12.844552 18244 server_base.cc:1034] running on GCE node
W20260926 04:02:12.844478 18252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:02:12.844491 18249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:02:12.844935 18244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:02:12.845120 18244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:02:12.846266 18244 hybrid_clock.cc:648] HybridClock initialized: now 1790395332846245 us; error 38 us; skew 500 ppm
I20260926 04:02:12.847429 18244 webserver.cc:460] Webserver started at http://127.0.98.3:43293/ using document root <none> and password file <none>
I20260926 04:02:12.847628 18244 fs_manager.cc:359] Metadata directory not provided
I20260926 04:02:12.847668 18244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:02:12.847754 18244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:02:12.848794 18244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395095474643-392-0/minicluster-data/ts-2/data/instance:
uuid: "f6374912b3b24f858d444abcd943f60a"
format_stamp: "Formatted at 2026-09-26 04:02:12 on dist-test-slave-nhpw"
server_key: "5c1878f9f5086cd9191de549dfac1cfe"
server_key_iv: "f9fabc87ad7677cde973f5e3e97197bf"
server_key_version: "encryptionkey@0"
I20260926 04:02:12.849093 18244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395095474643-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f6374912b3b24f858d444abcd943f60a"
format_stamp: "Formatted at 2026-09-26 04:02:12 on dist-test-slave-nhpw"
server_key: "5c1878f9f5086cd9191de549dfac1cfe"
server_key_iv: "f9fabc87ad7677cde973f5e3e97197bf"
server_key_version: "encryptionkey@0"
I20260926 04:02:12.850308 18244 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 04:02:12.850945 18258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:12.851121 18244 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:12.851218 18244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395095474643-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395095474643-392-0/minicluster-data/ts-2/wal
uuid: "f6374912b3b24f858d444abcd943f60a"
format_stamp: "Formatted at 2026-09-26 04:02:12 on dist-test-slave-nhpw"
server_key: "5c1878f9f5086cd9191de549dfac1cfe"
server_key_iv: "f9fabc87ad7677cde973f5e3e97197bf"
server_key_version: "encryptionkey@0"
I20260926 04:02:12.851298 18244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395095474643-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395095474643-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395095474643-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:02:12.872750 18244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:02:12.873068 18244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:02:12.873224 18244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:02:12.873463 18244 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:02:12.873782 18244 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:02:12.873836 18244 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:12.873870 18244 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:02:12.873922 18244 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:12.879460 18244 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46459
I20260926 04:02:12.879549 18371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46459 every 8 connection(s)
I20260926 04:02:12.879874 18244 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
I20260926 04:02:12.880822 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 18244
I20260926 04:02:12.880928 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395095474643-392-0/minicluster-data/ts-2/wal/instance
I20260926 04:02:12.881142 392 external_mini_cluster.cc:1428] Setting key 763252d3df2246f33337cf63f58636d4
I20260926 04:02:12.885404 17962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:12.880738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42709 (local address 127.0.98.62:33715)
0926 04:02:12.880817 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:12.880825 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:12.881029 (+ 204us) server_negotiation.cc:407] Connection header received
0926 04:02:12.881691 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:12.881692 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:12.881704 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:12.881748 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:12.882432 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:12.882936 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:12.883957 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:12.884168 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:12.884250 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:12.885146 (+ 896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:12.885148 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:12.885148 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:02:12.885153 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:12.885168 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:02:12.885172 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:12.885227 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:12.885300 (+ 73us) server_negotiation.cc:299] Negotiation successful
0926 04:02:12.885325 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 04:02:12.885630 18374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:12.880676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33715 (local address 127.0.98.3:42709)
0926 04:02:12.880889 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:12.880934 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 04:02:12.881603 (+ 669us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:02:12.881818 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:02:12.881822 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:02:12.881832 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:02:12.882334 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:12.882338 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:12.883061 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:12.883062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:12.883858 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:12.883860 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:12.884258 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:12.884259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:12.884404 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:12.885001 (+ 597us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 04:02:12.885002 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:02:12.885028 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 04:02:12.885052 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:02:12.885230 (+ 178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:02:12.885234 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:02:12.885237 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 04:02:12.885302 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 04:02:12.885401 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":98,"threads_started":1}
I20260926 04:02:12.886065 18372 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33715
I20260926 04:02:12.886154 18372 heartbeater.cc:464] Registering TS with master...
I20260926 04:02:12.886291 18372 heartbeater.cc:511] Master 127.0.98.62:33715 requested a full tablet report, sending...
I20260926 04:02:12.886637 17922 ts_manager.cc:194] Registered new tserver with Master: f6374912b3b24f858d444abcd943f60a (127.0.98.3:46459)
I20260926 04:02:12.887007 17922 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:42709
I20260926 04:02:12.893180 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 04:02:12.897156 17962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:12.894489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44096 (local address 127.0.98.62:33715)
0926 04:02:12.894555 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:12.894557 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:12.894697 (+ 140us) server_negotiation.cc:407] Connection header received
0926 04:02:12.894741 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:12.894742 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:12.894750 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:12.894798 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:12.894951 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:12.895398 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:12.896097 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:12.896265 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:12.896314 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:12.896931 (+ 617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:12.896932 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:12.896932 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:02:12.896936 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:12.896949 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:02:12.896952 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:12.897000 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:12.897047 (+ 47us) server_negotiation.cc:299] Negotiation successful
0926 04:02:12.897065 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 04:02:12.898607 17922 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44096:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 04:02:12.900141 17922 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 04:02:12.906011 18381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:12.902245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45905 (local address 127.0.0.1:58228)
0926 04:02:12.902429 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:12.902470 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 04:02:12.902537 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:02:12.902822 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:02:12.902825 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:02:12.902827 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:02:12.902890 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:12.902891 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:12.903864 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:12.903865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:12.904868 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:12.904871 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:12.905731 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:12.905732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:12.905799 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:12.905811 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 04:02:12.905892 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 04:02:12.905942 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":51,"threads_started":1}
I20260926 04:02:12.906025 18384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:12.902447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58228 (local address 127.0.98.1:45905)
0926 04:02:12.902662 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:12.902672 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:12.902680 (+ 8us) server_negotiation.cc:407] Connection header received
0926 04:02:12.902711 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:12.902712 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:12.902723 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:12.902805 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:02:12.903008 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:12.903771 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:12.904961 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:12.905672 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:12.905734 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:12.905820 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:12.905906 (+ 86us) server_negotiation.cc:299] Negotiation successful
0926 04:02:12.905944 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":100,"threads_started":1}
I20260926 04:02:12.906711 18044 tablet_service.cc:1467] Processing CreateTablet for tablet adabbf83f8b24babbca6039b7d66ad54 (DEFAULT_TABLE table=test-table [id=8cc3760feb084de6a1847fb5723b9e4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:02:12.906797 18383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:12.902283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46459 (local address 127.0.0.1:44862)
0926 04:02:12.902555 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:12.902566 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 04:02:12.902615 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:02:12.903156 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:02:12.903157 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:02:12.903159 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:02:12.903201 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:12.903202 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:12.904449 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:12.904450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:12.905581 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:12.905583 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:12.906563 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:12.906564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:12.906624 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:12.906628 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 04:02:12.906689 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 04:02:12.906725 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"mutex_wait_us":33,"thread_start_us":92,"threads_started":1}
I20260926 04:02:12.906813 18382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:12.902355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44862 (local address 127.0.98.3:46459)
0926 04:02:12.902990 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:12.902993 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:12.903003 (+ 10us) server_negotiation.cc:407] Connection header received
0926 04:02:12.903040 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:12.903042 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:12.903057 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:12.903272 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:02:12.903322 (+ 50us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:12.904363 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:12.905667 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:12.906496 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:12.906561 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:12.906634 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:12.906702 (+ 68us) server_negotiation.cc:299] Negotiation successful
0926 04:02:12.906738 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":597,"thread_start_us":74,"threads_started":1}
I20260926 04:02:12.907068 18044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adabbf83f8b24babbca6039b7d66ad54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:02:12.909130 18380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:12.902443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35187 (local address 127.0.0.1:56210)
0926 04:02:12.902807 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:12.902816 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 04:02:12.902866 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:02:12.903119 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:02:12.903120 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:02:12.903122 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:02:12.903157 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:12.903158 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:12.904442 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:12.904444 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:12.905448 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:12.905451 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:12.908904 (+ 3453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:12.908905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:12.908967 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:12.908971 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 04:02:12.909029 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 04:02:12.909063 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"mutex_wait_us":47,"thread_start_us":67,"threads_started":1}
I20260926 04:02:12.909158 18306 tablet_service.cc:1467] Processing CreateTablet for tablet adabbf83f8b24babbca6039b7d66ad54 (DEFAULT_TABLE table=test-table [id=8cc3760feb084de6a1847fb5723b9e4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:02:12.909435 18385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:12.902646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56210 (local address 127.0.98.2:35187)
0926 04:02:12.902789 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:12.902794 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:12.902921 (+ 127us) server_negotiation.cc:407] Connection header received
0926 04:02:12.902964 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:12.902967 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:12.902984 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:12.903056 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:02:12.903223 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:12.904338 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:12.905535 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:12.906280 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:12.906379 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:12.906460 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:12.909292 (+ 2832us) server_negotiation.cc:299] Negotiation successful
0926 04:02:12.909344 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":76,"threads_started":1}
I20260926 04:02:12.909533 18306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adabbf83f8b24babbca6039b7d66ad54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:02:12.910023 18386 tablet_bootstrap.cc:492] T adabbf83f8b24babbca6039b7d66ad54 P 13c199659db54660ad5cf0c3151f7046: Bootstrap starting.
I20260926 04:02:12.910055 18175 tablet_service.cc:1467] Processing CreateTablet for tablet adabbf83f8b24babbca6039b7d66ad54 (DEFAULT_TABLE table=test-table [id=8cc3760feb084de6a1847fb5723b9e4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:02:12.910375 18175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet adabbf83f8b24babbca6039b7d66ad54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:02:12.910667 18386 tablet_bootstrap.cc:654] T adabbf83f8b24babbca6039b7d66ad54 P 13c199659db54660ad5cf0c3151f7046: Neither blocks nor log segments found. Creating new log.
I20260926 04:02:12.910949 18386 log.cc:824] T adabbf83f8b24babbca6039b7d66ad54 P 13c199659db54660ad5cf0c3151f7046: Log is configured to *not* fsync() on all Append() calls
I20260926 04:02:12.911617 18386 tablet_bootstrap.cc:492] T adabbf83f8b24babbca6039b7d66ad54 P 13c199659db54660ad5cf0c3151f7046: No bootstrap required, opened a new log
I20260926 04:02:12.911698 18386 ts_tablet_manager.cc:1397] T adabbf83f8b24babbca6039b7d66ad54 P 13c199659db54660ad5cf0c3151f7046: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 04:02:12.912804 18388 tablet_bootstrap.cc:492] T adabbf83f8b24babbca6039b7d66ad54 P f6374912b3b24f858d444abcd943f60a: Bootstrap starting.
I20260926 04:02:12.913487 18388 tablet_bootstrap.cc:654] T adabbf83f8b24babbca6039b7d66ad54 P f6374912b3b24f858d444abcd943f60a: Neither blocks nor log segments found. Creating new log.
I20260926 04:02:12.913609 18389 tablet_bootstrap.cc:492] T adabbf83f8b24babbca6039b7d66ad54 P aed2e00ab7e94d2786e20459857979c0: Bootstrap starting.
I20260926 04:02:12.913923 18388 log.cc:824] T adabbf83f8b24babbca6039b7d66ad54 P f6374912b3b24f858d444abcd943f60a: Log is configured to *not* fsync() on all Append() calls
I20260926 04:02:12.914186 18389 tablet_bootstrap.cc:654] T adabbf83f8b24babbca6039b7d66ad54 P aed2e00ab7e94d2786e20459857979c0: Neither blocks nor log segments found. Creating new log.
I20260926 04:02:12.914187 18386 raft_consensus.cc:348] T adabbf83f8b24babbca6039b7d66ad54 P 13c199659db54660ad5cf0c3151f7046 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c199659db54660ad5cf0c3151f7046" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45905 } } peers { permanent_uuid: "aed2e00ab7e94d2786e20459857979c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35187 } } peers { permanent_uuid: "f6374912b3b24f858d444abcd943f60a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46459 } }
I20260926 04:02:12.914367 18386 raft_consensus.cc:374] T adabbf83f8b24babbca6039b7d66ad54 P 13c199659db54660ad5cf0c3151f7046 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:02:12.914415 18386 raft_consensus.cc:729] T adabbf83f8b24babbca6039b7d66ad54 P 13c199659db54660ad5cf0c3151f7046 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13c199659db54660ad5cf0c3151f7046, State: Initialized, Role: FOLLOWER
I20260926 04:02:12.914516 18389 log.cc:824] T adabbf83f8b24babbca6039b7d66ad54 P aed2e00ab7e94d2786e20459857979c0: Log is configured to *not* fsync() on all Append() calls
I20260926 04:02:12.914530 18386 consensus_queue.cc:260] T adabbf83f8b24babbca6039b7d66ad54 P 13c199659db54660ad5cf0c3151f7046 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c199659db54660ad5cf0c3151f7046" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45905 } } peers { permanent_uuid: "aed2e00ab7e94d2786e20459857979c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35187 } } peers { permanent_uuid: "f6374912b3b24f858d444abcd943f60a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46459 } }
I20260926 04:02:12.914620 18388 tablet_bootstrap.cc:492] T adabbf83f8b24babbca6039b7d66ad54 P f6374912b3b24f858d444abcd943f60a: No bootstrap required, opened a new log
I20260926 04:02:12.914688 18388 ts_tablet_manager.cc:1397] T adabbf83f8b24babbca6039b7d66ad54 P f6374912b3b24f858d444abcd943f60a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 04:02:12.914788 18386 ts_tablet_manager.cc:1428] T adabbf83f8b24babbca6039b7d66ad54 P 13c199659db54660ad5cf0c3151f7046: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 04:02:12.914920 18110 heartbeater.cc:503] Master 127.0.98.62:33715 was elected leader, sending a full tablet report...
I20260926 04:02:12.915658 18389 tablet_bootstrap.cc:492] T adabbf83f8b24babbca6039b7d66ad54 P aed2e00ab7e94d2786e20459857979c0: No bootstrap required, opened a new log
I20260926 04:02:12.915796 18389 ts_tablet_manager.cc:1397] T adabbf83f8b24babbca6039b7d66ad54 P aed2e00ab7e94d2786e20459857979c0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 04:02:12.916921 18388 raft_consensus.cc:348] T adabbf83f8b24babbca6039b7d66ad54 P f6374912b3b24f858d444abcd943f60a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c199659db54660ad5cf0c3151f7046" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45905 } } peers { permanent_uuid: "aed2e00ab7e94d2786e20459857979c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35187 } } peers { permanent_uuid: "f6374912b3b24f858d444abcd943f60a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46459 } }
I20260926 04:02:12.917076 18388 raft_consensus.cc:374] T adabbf83f8b24babbca6039b7d66ad54 P f6374912b3b24f858d444abcd943f60a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:02:12.917121 18388 raft_consensus.cc:729] T adabbf83f8b24babbca6039b7d66ad54 P f6374912b3b24f858d444abcd943f60a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6374912b3b24f858d444abcd943f60a, State: Initialized, Role: FOLLOWER
I20260926 04:02:12.917240 18388 consensus_queue.cc:260] T adabbf83f8b24babbca6039b7d66ad54 P f6374912b3b24f858d444abcd943f60a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c199659db54660ad5cf0c3151f7046" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45905 } } peers { permanent_uuid: "aed2e00ab7e94d2786e20459857979c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35187 } } peers { permanent_uuid: "f6374912b3b24f858d444abcd943f60a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46459 } }
I20260926 04:02:12.917500 18388 ts_tablet_manager.cc:1428] T adabbf83f8b24babbca6039b7d66ad54 P f6374912b3b24f858d444abcd943f60a: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 04:02:12.917517 18372 heartbeater.cc:503] Master 127.0.98.62:33715 was elected leader, sending a full tablet report...
I20260926 04:02:12.918088 18389 raft_consensus.cc:348] T adabbf83f8b24babbca6039b7d66ad54 P aed2e00ab7e94d2786e20459857979c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c199659db54660ad5cf0c3151f7046" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45905 } } peers { permanent_uuid: "aed2e00ab7e94d2786e20459857979c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35187 } } peers { permanent_uuid: "f6374912b3b24f858d444abcd943f60a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46459 } }
I20260926 04:02:12.918217 18389 raft_consensus.cc:374] T adabbf83f8b24babbca6039b7d66ad54 P aed2e00ab7e94d2786e20459857979c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:02:12.918260 18389 raft_consensus.cc:729] T adabbf83f8b24babbca6039b7d66ad54 P aed2e00ab7e94d2786e20459857979c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aed2e00ab7e94d2786e20459857979c0, State: Initialized, Role: FOLLOWER
I20260926 04:02:12.918367 18389 consensus_queue.cc:260] T adabbf83f8b24babbca6039b7d66ad54 P aed2e00ab7e94d2786e20459857979c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c199659db54660ad5cf0c3151f7046" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45905 } } peers { permanent_uuid: "aed2e00ab7e94d2786e20459857979c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35187 } } peers { permanent_uuid: "f6374912b3b24f858d444abcd943f60a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46459 } }
I20260926 04:02:12.918596 18389 ts_tablet_manager.cc:1428] T adabbf83f8b24babbca6039b7d66ad54 P aed2e00ab7e94d2786e20459857979c0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 04:02:12.918666 18241 heartbeater.cc:503] Master 127.0.98.62:33715 was elected leader, sending a full tablet report...
W20260926 04:02:12.997810 18242 tablet.cc:2369] T adabbf83f8b24babbca6039b7d66ad54 P aed2e00ab7e94d2786e20459857979c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 04:02:13.127159 18111 tablet.cc:2369] T adabbf83f8b24babbca6039b7d66ad54 P 13c199659db54660ad5cf0c3151f7046: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 04:02:13.130437 18373 tablet.cc:2369] T adabbf83f8b24babbca6039b7d66ad54 P f6374912b3b24f858d444abcd943f60a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 04:02:13.170361 18391 raft_consensus.cc:482] T adabbf83f8b24babbca6039b7d66ad54 P 13c199659db54660ad5cf0c3151f7046 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 04:02:13.170598 18391 raft_consensus.cc:504] T adabbf83f8b24babbca6039b7d66ad54 P 13c199659db54660ad5cf0c3151f7046 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c199659db54660ad5cf0c3151f7046" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45905 } } peers { permanent_uuid: "aed2e00ab7e94d2786e20459857979c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35187 } } peers { permanent_uuid: "f6374912b3b24f858d444abcd943f60a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46459 } }
I20260926 04:02:13.171069 18391 leader_election.cc:288] T adabbf83f8b24babbca6039b7d66ad54 P 13c199659db54660ad5cf0c3151f7046 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aed2e00ab7e94d2786e20459857979c0 (127.0.98.2:35187), f6374912b3b24f858d444abcd943f60a (127.0.98.3:46459)
I20260926 04:02:13.174059 18385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:13.171387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56831 (local address 127.0.98.2:35187)
0926 04:02:13.171430 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:13.171432 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:13.171483 (+ 51us) server_negotiation.cc:407] Connection header received
0926 04:02:13.171560 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:13.171561 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:13.171570 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:13.171622 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:02:13.171803 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:13.172285 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:13.173331 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:13.173694 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:13.173774 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:13.173784 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:13.173953 (+ 169us) server_negotiation.cc:299] Negotiation successful
0926 04:02:13.173990 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 04:02:13.174064 18382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:13.171341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50317 (local address 127.0.98.3:46459)
0926 04:02:13.171395 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:13.171398 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:13.171493 (+ 95us) server_negotiation.cc:407] Connection header received
0926 04:02:13.171570 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:13.171571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:13.171581 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:13.171650 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:02:13.171815 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:13.172289 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:13.173342 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:13.173707 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:13.173790 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:13.173800 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:13.173956 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 04:02:13.173995 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 04:02:13.174079 18396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:13.171270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35187 (local address 127.0.98.1:56831)
0926 04:02:13.171425 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:13.171437 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 04:02:13.171502 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:02:13.171640 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:02:13.171641 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:02:13.171643 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:02:13.171723 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:13.171725 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:13.172368 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:13.172370 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:13.173233 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:13.173235 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:13.173775 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:13.173776 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:13.173863 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:13.173867 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 04:02:13.173934 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 04:02:13.173977 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":1,"thread_start_us":65,"threads_started":1}
I20260926 04:02:13.174093 18395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:13.171271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46459 (local address 127.0.98.1:50317)
0926 04:02:13.171420 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:13.171431 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 04:02:13.171510 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:02:13.171668 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:02:13.171668 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:02:13.171670 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:02:13.171742 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:13.171743 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:13.172375 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:13.172377 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:13.173240 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:13.173243 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:13.173807 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:13.173808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:13.173869 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:13.173873 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 04:02:13.173948 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 04:02:13.173989 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":39,"thread_start_us":87,"threads_started":1}
I20260926 04:02:13.174430 18195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adabbf83f8b24babbca6039b7d66ad54" candidate_uuid: "13c199659db54660ad5cf0c3151f7046" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aed2e00ab7e94d2786e20459857979c0" is_pre_election: true
I20260926 04:02:13.174599 18195 raft_consensus.cc:2454] T adabbf83f8b24babbca6039b7d66ad54 P aed2e00ab7e94d2786e20459857979c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13c199659db54660ad5cf0c3151f7046 in term 0.
I20260926 04:02:13.174646 18326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adabbf83f8b24babbca6039b7d66ad54" candidate_uuid: "13c199659db54660ad5cf0c3151f7046" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6374912b3b24f858d444abcd943f60a" is_pre_election: true
I20260926 04:02:13.174808 18326 raft_consensus.cc:2454] T adabbf83f8b24babbca6039b7d66ad54 P f6374912b3b24f858d444abcd943f60a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 13c199659db54660ad5cf0c3151f7046 in term 0.
I20260926 04:02:13.174875 17997 leader_election.cc:302] T adabbf83f8b24babbca6039b7d66ad54 P 13c199659db54660ad5cf0c3151f7046 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13c199659db54660ad5cf0c3151f7046, aed2e00ab7e94d2786e20459857979c0; no voters:
I20260926 04:02:13.175022 18391 raft_consensus.cc:2781] T adabbf83f8b24babbca6039b7d66ad54 P 13c199659db54660ad5cf0c3151f7046 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 04:02:13.175087 18391 raft_consensus.cc:482] T adabbf83f8b24babbca6039b7d66ad54 P 13c199659db54660ad5cf0c3151f7046 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 04:02:13.175122 18391 raft_consensus.cc:3037] T adabbf83f8b24babbca6039b7d66ad54 P 13c199659db54660ad5cf0c3151f7046 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:02:13.175880 18391 raft_consensus.cc:504] T adabbf83f8b24babbca6039b7d66ad54 P 13c199659db54660ad5cf0c3151f7046 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c199659db54660ad5cf0c3151f7046" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45905 } } peers { permanent_uuid: "aed2e00ab7e94d2786e20459857979c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35187 } } peers { permanent_uuid: "f6374912b3b24f858d444abcd943f60a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46459 } }
I20260926 04:02:13.176030 18391 leader_election.cc:288] T adabbf83f8b24babbca6039b7d66ad54 P 13c199659db54660ad5cf0c3151f7046 [CANDIDATE]: Term 1 election: Requested vote from peers aed2e00ab7e94d2786e20459857979c0 (127.0.98.2:35187), f6374912b3b24f858d444abcd943f60a (127.0.98.3:46459)
I20260926 04:02:13.176172 18326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adabbf83f8b24babbca6039b7d66ad54" candidate_uuid: "13c199659db54660ad5cf0c3151f7046" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f6374912b3b24f858d444abcd943f60a"
I20260926 04:02:13.176215 18195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "adabbf83f8b24babbca6039b7d66ad54" candidate_uuid: "13c199659db54660ad5cf0c3151f7046" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aed2e00ab7e94d2786e20459857979c0"
I20260926 04:02:13.176271 18326 raft_consensus.cc:3037] T adabbf83f8b24babbca6039b7d66ad54 P f6374912b3b24f858d444abcd943f60a [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:02:13.176285 18195 raft_consensus.cc:3037] T adabbf83f8b24babbca6039b7d66ad54 P aed2e00ab7e94d2786e20459857979c0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:02:13.177083 18195 raft_consensus.cc:2454] T adabbf83f8b24babbca6039b7d66ad54 P aed2e00ab7e94d2786e20459857979c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13c199659db54660ad5cf0c3151f7046 in term 1.
I20260926 04:02:13.177112 18326 raft_consensus.cc:2454] T adabbf83f8b24babbca6039b7d66ad54 P f6374912b3b24f858d444abcd943f60a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 13c199659db54660ad5cf0c3151f7046 in term 1.
I20260926 04:02:13.177277 17997 leader_election.cc:302] T adabbf83f8b24babbca6039b7d66ad54 P 13c199659db54660ad5cf0c3151f7046 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13c199659db54660ad5cf0c3151f7046, aed2e00ab7e94d2786e20459857979c0; no voters:
I20260926 04:02:13.177387 18391 raft_consensus.cc:2781] T adabbf83f8b24babbca6039b7d66ad54 P 13c199659db54660ad5cf0c3151f7046 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:02:13.177522 18391 raft_consensus.cc:686] T adabbf83f8b24babbca6039b7d66ad54 P 13c199659db54660ad5cf0c3151f7046 [term 1 LEADER]: Becoming Leader. State: Replica: 13c199659db54660ad5cf0c3151f7046, State: Running, Role: LEADER
I20260926 04:02:13.177615 18391 consensus_queue.cc:237] T adabbf83f8b24babbca6039b7d66ad54 P 13c199659db54660ad5cf0c3151f7046 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c199659db54660ad5cf0c3151f7046" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45905 } } peers { permanent_uuid: "aed2e00ab7e94d2786e20459857979c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35187 } } peers { permanent_uuid: "f6374912b3b24f858d444abcd943f60a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46459 } }
I20260926 04:02:13.178459 17922 catalog_manager.cc:5495] T adabbf83f8b24babbca6039b7d66ad54 P 13c199659db54660ad5cf0c3151f7046 reported cstate change: term changed from 0 to 1, leader changed from <none> to 13c199659db54660ad5cf0c3151f7046 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "13c199659db54660ad5cf0c3151f7046" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13c199659db54660ad5cf0c3151f7046" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45905 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aed2e00ab7e94d2786e20459857979c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f6374912b3b24f858d444abcd943f60a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46459 } health_report { overall_health: UNKNOWN } } }
I20260926 04:02:13.189452 18384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:13.187130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58238 (local address 127.0.98.1:45905)
0926 04:02:13.187196 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:13.187198 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:13.187206 (+ 8us) server_negotiation.cc:407] Connection header received
0926 04:02:13.187273 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:13.187274 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:13.187291 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:13.187351 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 04:02:13.187473 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:13.187996 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:13.188732 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:13.188952 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:13.189005 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:13.189150 (+ 145us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 04:02:13.189243 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 04:02:13.189300 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:13.189331 (+ 31us) server_negotiation.cc:299] Negotiation successful
0926 04:02:13.189347 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 04:02:13.191311 18195 raft_consensus.cc:1261] T adabbf83f8b24babbca6039b7d66ad54 P aed2e00ab7e94d2786e20459857979c0 [term 1 FOLLOWER]: Refusing update from remote peer 13c199659db54660ad5cf0c3151f7046: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:02:13.191316 18326 raft_consensus.cc:1261] T adabbf83f8b24babbca6039b7d66ad54 P f6374912b3b24f858d444abcd943f60a [term 1 FOLLOWER]: Refusing update from remote peer 13c199659db54660ad5cf0c3151f7046: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:02:13.191735 18391 consensus_queue.cc:1035] T adabbf83f8b24babbca6039b7d66ad54 P 13c199659db54660ad5cf0c3151f7046 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f6374912b3b24f858d444abcd943f60a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:02:13.191898 18397 consensus_queue.cc:1035] T adabbf83f8b24babbca6039b7d66ad54 P 13c199659db54660ad5cf0c3151f7046 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aed2e00ab7e94d2786e20459857979c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:02:13.194430 18400 mvcc.cc:205] Tried to move back new op lower bound from 7333459284749139968 to 7333459284696096768. Current Snapshot: MvccSnapshot[applied={T|T < 7333459284749139968}]
I20260926 04:02:13.196774 17921 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44096:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 04:02:13.196869 17921 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44096:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 04:02:13.197641 17921 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b9b760b328814479ba55d27e0d56da67: Sending DeleteTablet for 3 replicas of tablet adabbf83f8b24babbca6039b7d66ad54
I20260926 04:02:13.197999 18044 tablet_service.cc:1514] Processing DeleteTablet for tablet adabbf83f8b24babbca6039b7d66ad54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:02:13 UTC) from {username='slave'} at 127.0.0.1:58228
I20260926 04:02:13.198038 18175 tablet_service.cc:1514] Processing DeleteTablet for tablet adabbf83f8b24babbca6039b7d66ad54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:02:13 UTC) from {username='slave'} at 127.0.0.1:56210
I20260926 04:02:13.198163 18306 tablet_service.cc:1514] Processing DeleteTablet for tablet adabbf83f8b24babbca6039b7d66ad54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:02:13 UTC) from {username='slave'} at 127.0.0.1:44862
I20260926 04:02:13.198388 18411 tablet_replica.cc:330] T adabbf83f8b24babbca6039b7d66ad54 P f6374912b3b24f858d444abcd943f60a: stopping tablet replica
I20260926 04:02:13.198504 18411 raft_consensus.cc:2229] T adabbf83f8b24babbca6039b7d66ad54 P f6374912b3b24f858d444abcd943f60a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 04:02:13.198603 18411 raft_consensus.cc:2258] T adabbf83f8b24babbca6039b7d66ad54 P f6374912b3b24f858d444abcd943f60a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:02:13.198689 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 17982
I20260926 04:02:13.199589 18411 ts_tablet_manager.cc:1905] T adabbf83f8b24babbca6039b7d66ad54 P f6374912b3b24f858d444abcd943f60a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:02:13.201325 18411 ts_tablet_manager.cc:1918] T adabbf83f8b24babbca6039b7d66ad54 P f6374912b3b24f858d444abcd943f60a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 04:02:13.201457 18411 log.cc:1196] T adabbf83f8b24babbca6039b7d66ad54 P f6374912b3b24f858d444abcd943f60a: Deleting WAL directory at /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790395095474643-392-0/minicluster-data/ts-2/wal/wals/adabbf83f8b24babbca6039b7d66ad54
I20260926 04:02:13.201701 18411 ts_tablet_manager.cc:1939] T adabbf83f8b24babbca6039b7d66ad54 P f6374912b3b24f858d444abcd943f60a: Deleting consensus metadata
I20260926 04:02:13.202085 17908 catalog_manager.cc:4841] TS f6374912b3b24f858d444abcd943f60a (127.0.98.3:46459): tablet adabbf83f8b24babbca6039b7d66ad54 (table test-table [id=8cc3760feb084de6a1847fb5723b9e4d]) successfully deleted
W20260926 04:02:13.204619 17906 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45905 (error 108)
I20260926 04:02:13.204706 18410 tablet_replica.cc:330] T adabbf83f8b24babbca6039b7d66ad54 P aed2e00ab7e94d2786e20459857979c0: stopping tablet replica
I20260926 04:02:13.204824 18410 raft_consensus.cc:2229] T adabbf83f8b24babbca6039b7d66ad54 P aed2e00ab7e94d2786e20459857979c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 04:02:13.204937 18410 raft_consensus.cc:2258] T adabbf83f8b24babbca6039b7d66ad54 P aed2e00ab7e94d2786e20459857979c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:02:13.204946 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 18113
I20260926 04:02:13.205837 18380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:13.205706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45905 (local address 127.0.0.1:58252)
0926 04:02:13.205756 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:13.205794 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45905: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 04:02:13.205996 17906 catalog_manager.cc:4584] TS 13c199659db54660ad5cf0c3151f7046 (127.0.98.1:45905): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet adabbf83f8b24babbca6039b7d66ad54: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45905: connect: Connection refused (error 111)
I20260926 04:02:13.209843 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 18244
I20260926 04:02:13.209914 18380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:13.209743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35187 (local address 127.0.0.1:56226)
0926 04:02:13.209805 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:13.209836 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:35187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260926 04:02:13.214627 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 17891
2026-09-26T04:02:13Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (895 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 04:02:13 dist-test-slave-nhpw krb5kdc[18415](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 04:02:13 dist-test-slave-nhpw krb5kdc[18415](info): set up 2 sockets
Sep 26 04:02:13 dist-test-slave-nhpw krb5kdc[18415](info): commencing operation
krb5kdc: starting...
W20260926 04:02:14.252377 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/krb5kdc/test-user.keytab.
Sep 26 04:02:14 dist-test-slave-nhpw krb5kdc[18415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T04:02:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:02:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 04:02:14.363822 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41111
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41111
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:02:14.443254 18428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:02:14.443451 18428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:02:14.443473 18428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:02:14.444623 18428 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:02:14.444690 18428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:02:14.444705 18428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:02:14.444717 18428 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:02:14.444730 18428 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:02:14.446213 18428 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18428
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:02:14.446460 18428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:02:14.446686 18428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:02:14.451521 18436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:02:14.451531 18434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:02:14.451539 18433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:02:14.451782 18428 server_base.cc:1034] running on GCE node
I20260926 04:02:14.452080 18428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:02:14.452322 18428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:02:14.453459 18428 hybrid_clock.cc:648] HybridClock initialized: now 1790395334453432 us; error 43 us; skew 500 ppm
Sep 26 04:02:14 dist-test-slave-nhpw krb5kdc[18415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395334, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:02:14.455814 18428 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 04:02:14.456270 18428 webserver.cc:460] Webserver started at http://127.0.98.62:45449/ using document root <none> and password file <none>
I20260926 04:02:14.456506 18428 fs_manager.cc:359] Metadata directory not provided
I20260926 04:02:14.456557 18428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:02:14.456738 18428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:02:14.457790 18428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/minicluster-data/master-0/data/instance:
uuid: "8b90b5633d314eb69c8f6b00bb97ea67"
format_stamp: "Formatted at 2026-09-26 04:02:14 on dist-test-slave-nhpw"
server_key: "0230f97eb3f922a53a0286508a9a717e"
server_key_iv: "320b14778e34f5eb057d649fa68595fb"
server_key_version: "encryptionkey@0"
I20260926 04:02:14.458163 18428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/minicluster-data/master-0/wal/instance:
uuid: "8b90b5633d314eb69c8f6b00bb97ea67"
format_stamp: "Formatted at 2026-09-26 04:02:14 on dist-test-slave-nhpw"
server_key: "0230f97eb3f922a53a0286508a9a717e"
server_key_iv: "320b14778e34f5eb057d649fa68595fb"
server_key_version: "encryptionkey@0"
I20260926 04:02:14.459446 18428 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 04:02:14.460130 18443 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:14.460299 18428 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 04:02:14.460389 18428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/minicluster-data/master-0/wal
uuid: "8b90b5633d314eb69c8f6b00bb97ea67"
format_stamp: "Formatted at 2026-09-26 04:02:14 on dist-test-slave-nhpw"
server_key: "0230f97eb3f922a53a0286508a9a717e"
server_key_iv: "320b14778e34f5eb057d649fa68595fb"
server_key_version: "encryptionkey@0"
I20260926 04:02:14.460469 18428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:02:14.481053 18428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:02:14.482975 18428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:02:14.483157 18428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:02:14.487265 18428 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41111
I20260926 04:02:14.487303 18495 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41111 every 8 connection(s)
I20260926 04:02:14.487674 18428 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
I20260926 04:02:14.488077 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 18428
I20260926 04:02:14.488210 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/minicluster-data/master-0/wal/instance
I20260926 04:02:14.488428 392 external_mini_cluster.cc:1428] Setting key 281ad35499d3088f1028ac7aa0b05b54
I20260926 04:02:14.488864 18496 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:02:14.491567 18496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b90b5633d314eb69c8f6b00bb97ea67: Bootstrap starting.
I20260926 04:02:14.492165 18496 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b90b5633d314eb69c8f6b00bb97ea67: Neither blocks nor log segments found. Creating new log.
I20260926 04:02:14.492443 18496 log.cc:824] T 00000000000000000000000000000000 P 8b90b5633d314eb69c8f6b00bb97ea67: Log is configured to *not* fsync() on all Append() calls
I20260926 04:02:14.493135 18496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b90b5633d314eb69c8f6b00bb97ea67: No bootstrap required, opened a new log
I20260926 04:02:14.495405 18496 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b90b5633d314eb69c8f6b00bb97ea67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b90b5633d314eb69c8f6b00bb97ea67" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41111 } }
I20260926 04:02:14.495591 18496 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b90b5633d314eb69c8f6b00bb97ea67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:02:14.495627 18496 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b90b5633d314eb69c8f6b00bb97ea67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b90b5633d314eb69c8f6b00bb97ea67, State: Initialized, Role: FOLLOWER
Sep 26 04:02:14 dist-test-slave-nhpw krb5kdc[18415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 04:02:14.495774 18496 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b90b5633d314eb69c8f6b00bb97ea67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b90b5633d314eb69c8f6b00bb97ea67" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41111 } }
I20260926 04:02:14.495863 18496 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b90b5633d314eb69c8f6b00bb97ea67 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:02:14.495900 18496 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b90b5633d314eb69c8f6b00bb97ea67 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:02:14.495950 18496 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b90b5633d314eb69c8f6b00bb97ea67 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:02:14.496593 18496 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b90b5633d314eb69c8f6b00bb97ea67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b90b5633d314eb69c8f6b00bb97ea67" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41111 } }
I20260926 04:02:14.496721 18496 leader_election.cc:302] T 00000000000000000000000000000000 P 8b90b5633d314eb69c8f6b00bb97ea67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b90b5633d314eb69c8f6b00bb97ea67; no voters:
I20260926 04:02:14.496901 18496 leader_election.cc:288] T 00000000000000000000000000000000 P 8b90b5633d314eb69c8f6b00bb97ea67 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 04:02:14.496946 18501 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b90b5633d314eb69c8f6b00bb97ea67 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:02:14.497102 18501 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b90b5633d314eb69c8f6b00bb97ea67 [term 1 LEADER]: Becoming Leader. State: Replica: 8b90b5633d314eb69c8f6b00bb97ea67, State: Running, Role: LEADER
I20260926 04:02:14.497203 18496 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b90b5633d314eb69c8f6b00bb97ea67 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:02:14.497195 18501 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b90b5633d314eb69c8f6b00bb97ea67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b90b5633d314eb69c8f6b00bb97ea67" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41111 } }
I20260926 04:02:14.497598 18501 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b90b5633d314eb69c8f6b00bb97ea67 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b90b5633d314eb69c8f6b00bb97ea67. Latest consensus state: current_term: 1 leader_uuid: "8b90b5633d314eb69c8f6b00bb97ea67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b90b5633d314eb69c8f6b00bb97ea67" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41111 } } }
I20260926 04:02:14.497684 18501 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b90b5633d314eb69c8f6b00bb97ea67 [sys.catalog]: This master's current role is: LEADER
I20260926 04:02:14.497944 18505 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:02:14.498052 18503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b90b5633d314eb69c8f6b00bb97ea67 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b90b5633d314eb69c8f6b00bb97ea67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b90b5633d314eb69c8f6b00bb97ea67" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41111 } } }
I20260926 04:02:14.498193 18503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b90b5633d314eb69c8f6b00bb97ea67 [sys.catalog]: This master's current role is: LEADER
I20260926 04:02:14.498477 18505 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:02:14.500883 18499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:14.489104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37246 (local address 127.0.98.62:41111)
0926 04:02:14.489301 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:14.489310 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:14.489343 (+ 33us) server_negotiation.cc:407] Connection header received
0926 04:02:14.490237 (+ 894us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:14.490245 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:14.490298 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:14.490478 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:14.491141 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:14.492132 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:14.493068 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:14.493567 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:14.493686 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:14.496278 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:14.496283 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:14.496283 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:02:14.496294 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:14.499131 (+ 2837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:14.499532 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:14.499535 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:14.499535 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:14.499638 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:14.499850 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:14.499851 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:14.499851 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:14.500089 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:14.500246 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:14.500522 (+ 276us) server_negotiation.cc:299] Negotiation successful
0926 04:02:14.500684 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":86,"threads_started":1}
I20260926 04:02:14.501173 18505 catalog_manager.cc:1347] Generated new cluster ID: 4eb1c6538fb948ecb329796684522f0f
I20260926 04:02:14.501224 18505 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 04:02:14.511556 18505 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:02:14.512282 18505 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:02:14.518695 18505 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b90b5633d314eb69c8f6b00bb97ea67: Generated new TSK 0
I20260926 04:02:14.518896 18505 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 04:02:14.559437 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41111
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:02:14.675519 18522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:02:14.675774 18522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:02:14.675807 18522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:02:14.677572 18522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:02:14.677632 18522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:02:14.677668 18522 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 04:02:14.679633 18522 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18522
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:02:14.679880 18522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:02:14.680121 18522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:02:14.685282 18527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:02:14.685271 18530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:02:14.685271 18528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:02:14.685586 18522 server_base.cc:1034] running on GCE node
I20260926 04:02:14.685777 18522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:02:14.685958 18522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:02:14.687119 18522 hybrid_clock.cc:648] HybridClock initialized: now 1790395334687111 us; error 39 us; skew 500 ppm
Sep 26 04:02:14 dist-test-slave-nhpw krb5kdc[18415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395334, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:02:14.689662 18522 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 04:02:14.690023 18522 webserver.cc:460] Webserver started at http://127.0.98.1:36231/ using document root <none> and password file <none>
I20260926 04:02:14.690256 18522 fs_manager.cc:359] Metadata directory not provided
I20260926 04:02:14.690302 18522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:02:14.690385 18522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:02:14.691332 18522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/minicluster-data/ts-0/data/instance:
uuid: "1f853c762d8a4074acf9bd2bb54f1c21"
format_stamp: "Formatted at 2026-09-26 04:02:14 on dist-test-slave-nhpw"
server_key: "ed2ec63bbeb5d96db2757da4cd84dfc2"
server_key_iv: "22dc10f327f82e7bfdd3045e11259472"
server_key_version: "encryptionkey@0"
I20260926 04:02:14.691622 18522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1f853c762d8a4074acf9bd2bb54f1c21"
format_stamp: "Formatted at 2026-09-26 04:02:14 on dist-test-slave-nhpw"
server_key: "ed2ec63bbeb5d96db2757da4cd84dfc2"
server_key_iv: "22dc10f327f82e7bfdd3045e11259472"
server_key_version: "encryptionkey@0"
I20260926 04:02:14.692874 18522 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 04:02:14.693497 18537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:14.693642 18522 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 04:02:14.693760 18522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/minicluster-data/ts-0/wal
uuid: "1f853c762d8a4074acf9bd2bb54f1c21"
format_stamp: "Formatted at 2026-09-26 04:02:14 on dist-test-slave-nhpw"
server_key: "ed2ec63bbeb5d96db2757da4cd84dfc2"
server_key_iv: "22dc10f327f82e7bfdd3045e11259472"
server_key_version: "encryptionkey@0"
I20260926 04:02:14.693859 18522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:02:14.704022 18522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:02:14.706197 18522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:02:14.706306 18522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:02:14.706461 18522 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:02:14.706753 18522 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:02:14.706785 18522 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:14.706807 18522 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:02:14.706852 18522 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:14.712344 18522 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39061
I20260926 04:02:14.712417 18650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39061 every 8 connection(s)
I20260926 04:02:14.712814 18522 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
I20260926 04:02:14.714948 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 18522
I20260926 04:02:14.715036 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/minicluster-data/ts-0/wal/instance
I20260926 04:02:14.715270 392 external_mini_cluster.cc:1428] Setting key c704ec11949ff347985f578ee7aef5e8
Sep 26 04:02:14 dist-test-slave-nhpw krb5kdc[18415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395334, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 04:02:14.722842 18499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:14.713525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41321 (local address 127.0.98.62:41111)
0926 04:02:14.713595 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:14.713597 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:14.713807 (+ 210us) server_negotiation.cc:407] Connection header received
0926 04:02:14.714694 (+ 887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:14.714695 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:14.714705 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:14.714761 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:14.715454 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:14.716169 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:14.717253 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:14.717476 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:14.717570 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:14.719866 (+ 2296us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:14.719868 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:14.719868 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:02:14.719876 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:14.721550 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:14.721911 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:14.721913 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:14.721913 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:14.721977 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:14.722196 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:14.722197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:14.722198 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:14.722342 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:14.722418 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:14.722653 (+ 235us) server_negotiation.cc:299] Negotiation successful
0926 04:02:14.722740 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 04:02:14.722865 18653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:14.713506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41111 (local address 127.0.98.1:41321)
0926 04:02:14.713687 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:14.713726 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 04:02:14.714608 (+ 882us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:02:14.714815 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:02:14.714818 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:02:14.715128 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:02:14.715348 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:14.715354 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:14.716257 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:14.716258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:14.717163 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:14.717166 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:14.717583 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:14.717584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:14.717711 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:14.718253 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:02:14.718263 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:02:14.719779 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:02:14.721629 (+ 1850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:02:14.721635 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:02:14.721636 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:02:14.721839 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:02:14.722044 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:02:14.722045 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:02:14.722045 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:02:14.722135 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:02:14.722411 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:02:14.722414 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:02:14.722519 (+ 105us) client_negotiation.cc:769] Sending connection context
0926 04:02:14.722606 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 04:02:14.722716 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":107,"threads_started":1}
I20260926 04:02:14.723660 18651 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41111
I20260926 04:02:14.723740 18651 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 04:02:14.723928 18651 heartbeater.cc:511] Master 127.0.98.62:41111 requested a full tablet report, sending...
I20260926 04:02:14.724481 18460 ts_manager.cc:194] Registered new tserver with Master: 1f853c762d8a4074acf9bd2bb54f1c21 (127.0.98.1:39061)
I20260926 04:02:14.725172 18460 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:41321
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 04:02:14.739835 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41111
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:02:14.828188 18656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:02:14.828380 18656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:02:14.828403 18656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:02:14.829505 18656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:02:14.829542 18656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:02:14.829567 18656 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 04:02:14.830964 18656 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18656
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:02:14.831147 18656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:02:14.831316 18656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:02:14.836169 18661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:02:14.836247 18664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:02:14.836225 18656 server_base.cc:1034] running on GCE node
W20260926 04:02:14.836411 18662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:02:14.836817 18656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:02:14.837069 18656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:02:14.838218 18656 hybrid_clock.cc:648] HybridClock initialized: now 1790395334838205 us; error 27 us; skew 500 ppm
Sep 26 04:02:14 dist-test-slave-nhpw krb5kdc[18415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395334, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:02:14.840694 18656 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 04:02:14.841173 18656 webserver.cc:460] Webserver started at http://127.0.98.2:39731/ using document root <none> and password file <none>
I20260926 04:02:14.841408 18656 fs_manager.cc:359] Metadata directory not provided
I20260926 04:02:14.841476 18656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:02:14.841629 18656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:02:14.842658 18656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/minicluster-data/ts-1/data/instance:
uuid: "c10aa3ad615f49219ffdcfbda80e603d"
format_stamp: "Formatted at 2026-09-26 04:02:14 on dist-test-slave-nhpw"
server_key: "1ab750ae964fcbfac8dce498a5bf7708"
server_key_iv: "11e3daaa71553a6cf62530b8b3675821"
server_key_version: "encryptionkey@0"
I20260926 04:02:14.843032 18656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c10aa3ad615f49219ffdcfbda80e603d"
format_stamp: "Formatted at 2026-09-26 04:02:14 on dist-test-slave-nhpw"
server_key: "1ab750ae964fcbfac8dce498a5bf7708"
server_key_iv: "11e3daaa71553a6cf62530b8b3675821"
server_key_version: "encryptionkey@0"
I20260926 04:02:14.844329 18656 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 04:02:14.845119 18671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:14.845274 18656 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:14.845374 18656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/minicluster-data/ts-1/wal
uuid: "c10aa3ad615f49219ffdcfbda80e603d"
format_stamp: "Formatted at 2026-09-26 04:02:14 on dist-test-slave-nhpw"
server_key: "1ab750ae964fcbfac8dce498a5bf7708"
server_key_iv: "11e3daaa71553a6cf62530b8b3675821"
server_key_version: "encryptionkey@0"
I20260926 04:02:14.845455 18656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:02:14.876154 18656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:02:14.878348 18656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:02:14.878520 18656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:02:14.878777 18656 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:02:14.879101 18656 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:02:14.879155 18656 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:14.879200 18656 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:02:14.879226 18656 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:14.885499 18656 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36689
I20260926 04:02:14.885555 18784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36689 every 8 connection(s)
I20260926 04:02:14.885874 18656 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
Sep 26 04:02:14 dist-test-slave-nhpw krb5kdc[18415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395334, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 04:02:14.894585 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 18656
I20260926 04:02:14.894719 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/minicluster-data/ts-1/wal/instance
I20260926 04:02:14.894910 392 external_mini_cluster.cc:1428] Setting key 309d7a84bc65e1d0e2f6ceb28f955d22
I20260926 04:02:14.895345 18499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:14.886635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49481 (local address 127.0.98.62:41111)
0926 04:02:14.886698 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:14.886700 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:14.886917 (+ 217us) server_negotiation.cc:407] Connection header received
0926 04:02:14.887845 (+ 928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:14.887846 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:14.887855 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:14.887916 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:14.888520 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:14.889105 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:14.890066 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:14.890246 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:14.890328 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:14.892370 (+ 2042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:14.892371 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:14.892371 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:02:14.892378 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:14.893838 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:14.894289 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:14.894290 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:14.894290 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:14.894340 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:14.894600 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:14.894602 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:14.894602 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:14.894784 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:14.894855 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:14.895166 (+ 311us) server_negotiation.cc:299] Negotiation successful
0926 04:02:14.895253 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 04:02:14.895547 18787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:14.886625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41111 (local address 127.0.98.2:49481)
0926 04:02:14.886790 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:14.886832 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 04:02:14.887758 (+ 926us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:02:14.887973 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:02:14.887976 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:02:14.888243 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:02:14.888438 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:14.888442 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:14.889194 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:14.889195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:14.889990 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:14.889992 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:14.890332 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:14.890333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:14.890433 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:14.890881 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:02:14.890889 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:02:14.892282 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:02:14.893914 (+ 1632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:02:14.893920 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:02:14.893921 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:02:14.894235 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:02:14.894403 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:02:14.894403 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:02:14.894404 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:02:14.894513 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:02:14.894843 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:02:14.894849 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:02:14.895008 (+ 159us) client_negotiation.cc:769] Sending connection context
0926 04:02:14.895131 (+ 123us) client_negotiation.cc:240] Negotiation successful
0926 04:02:14.895294 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":94,"threads_started":1}
I20260926 04:02:14.895994 18785 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41111
I20260926 04:02:14.896095 18785 heartbeater.cc:464] Registering TS with master...
I20260926 04:02:14.896288 18785 heartbeater.cc:511] Master 127.0.98.62:41111 requested a full tablet report, sending...
I20260926 04:02:14.896703 18460 ts_manager.cc:194] Registered new tserver with Master: c10aa3ad615f49219ffdcfbda80e603d (127.0.98.2:36689)
I20260926 04:02:14.897275 18460 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49481
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 04:02:14.919695 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41111
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:02:15.001183 18790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:02:15.001400 18790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:02:15.001438 18790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:02:15.002610 18790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:02:15.002674 18790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:02:15.002719 18790 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 04:02:15.004292 18790 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:02:15.004562 18790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:02:15.004806 18790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:02:15.010216 18798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:02:15.010206 18796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:02:15.010202 18795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:02:15.010237 18790 server_base.cc:1034] running on GCE node
I20260926 04:02:15.010614 18790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:02:15.010916 18790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:02:15.012112 18790 hybrid_clock.cc:648] HybridClock initialized: now 1790395335012098 us; error 30 us; skew 500 ppm
Sep 26 04:02:15 dist-test-slave-nhpw krb5kdc[18415](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395335, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:02:15.014691 18790 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 04:02:15.015156 18790 webserver.cc:460] Webserver started at http://127.0.98.3:43789/ using document root <none> and password file <none>
I20260926 04:02:15.015409 18790 fs_manager.cc:359] Metadata directory not provided
I20260926 04:02:15.015501 18790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:02:15.015642 18790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:02:15.016832 18790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/minicluster-data/ts-2/data/instance:
uuid: "ebdda04d615b458eaa36d7022778dc22"
format_stamp: "Formatted at 2026-09-26 04:02:15 on dist-test-slave-nhpw"
server_key: "5f54e012c8045af99fd82bcef54921ee"
server_key_iv: "0a204e0a5b2af7131434e7fd69f7d99e"
server_key_version: "encryptionkey@0"
I20260926 04:02:15.017206 18790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ebdda04d615b458eaa36d7022778dc22"
format_stamp: "Formatted at 2026-09-26 04:02:15 on dist-test-slave-nhpw"
server_key: "5f54e012c8045af99fd82bcef54921ee"
server_key_iv: "0a204e0a5b2af7131434e7fd69f7d99e"
server_key_version: "encryptionkey@0"
I20260926 04:02:15.018555 18790 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 04:02:15.019366 18805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:15.019527 18790 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 04:02:15.019626 18790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/minicluster-data/ts-2/wal
uuid: "ebdda04d615b458eaa36d7022778dc22"
format_stamp: "Formatted at 2026-09-26 04:02:15 on dist-test-slave-nhpw"
server_key: "5f54e012c8045af99fd82bcef54921ee"
server_key_iv: "0a204e0a5b2af7131434e7fd69f7d99e"
server_key_version: "encryptionkey@0"
I20260926 04:02:15.019711 18790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:02:15.030889 18790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:02:15.032624 18790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:02:15.032836 18790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:02:15.033066 18790 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:02:15.033372 18790 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:02:15.033427 18790 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:15.033460 18790 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:02:15.033488 18790 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:15.039105 18790 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34447
I20260926 04:02:15.039173 18918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34447 every 8 connection(s)
I20260926 04:02:15.039496 18790 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
I20260926 04:02:15.044281 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 18790
I20260926 04:02:15.044399 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/minicluster-data/ts-2/wal/instance
I20260926 04:02:15.044620 392 external_mini_cluster.cc:1428] Setting key 757eca38e22e70d3b5f201e4df630bc4
Sep 26 04:02:15 dist-test-slave-nhpw krb5kdc[18415](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395335, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 04:02:15.049592 18499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:15.040196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52383 (local address 127.0.98.62:41111)
0926 04:02:15.040238 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:15.040240 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:15.040440 (+ 200us) server_negotiation.cc:407] Connection header received
0926 04:02:15.041468 (+ 1028us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:15.041469 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:15.041480 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:15.041546 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:15.042133 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:15.042624 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:15.043591 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:15.043782 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:15.043850 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:15.046459 (+ 2609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:15.046461 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:15.046461 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:02:15.046469 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:15.047946 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:15.048539 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:15.048541 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:15.048541 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:15.048604 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:15.048926 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:15.048927 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:15.048928 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:15.049046 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:15.049126 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:15.049409 (+ 283us) server_negotiation.cc:299] Negotiation successful
0926 04:02:15.049503 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 04:02:15.049772 18921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:15.040158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41111 (local address 127.0.98.3:52383)
0926 04:02:15.040326 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:15.040368 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 04:02:15.041351 (+ 983us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:02:15.041619 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:02:15.041623 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:02:15.041867 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:02:15.042056 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:15.042061 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:15.042699 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:15.042700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:15.043492 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:15.043493 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:15.043865 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:15.043865 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:15.043964 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:15.044596 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:02:15.044607 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:02:15.046342 (+ 1735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:02:15.048125 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:02:15.048131 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:02:15.048132 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:02:15.048436 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:02:15.048718 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:02:15.048719 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:02:15.048719 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:02:15.048836 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:02:15.049117 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:02:15.049124 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:02:15.049268 (+ 144us) client_negotiation.cc:769] Sending connection context
0926 04:02:15.049378 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 04:02:15.049533 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":93,"threads_started":1}
I20260926 04:02:15.050169 18919 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41111
I20260926 04:02:15.050259 18919 heartbeater.cc:464] Registering TS with master...
I20260926 04:02:15.050400 18919 heartbeater.cc:511] Master 127.0.98.62:41111 requested a full tablet report, sending...
I20260926 04:02:15.050751 18460 ts_manager.cc:194] Registered new tserver with Master: ebdda04d615b458eaa36d7022778dc22 (127.0.98.3:34447)
I20260926 04:02:15.051091 18460 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52383
I20260926 04:02:15.056833 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 04:02:15.064841 18499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:15.058214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37260 (local address 127.0.98.62:41111)
0926 04:02:15.058253 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:15.058255 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:15.058377 (+ 122us) server_negotiation.cc:407] Connection header received
0926 04:02:15.058455 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:15.058456 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:15.058466 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:15.058516 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:15.059092 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:15.059582 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:15.060326 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:15.060517 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:15.060587 (+ 70us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:15.061975 (+ 1388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:15.061976 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:15.061977 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:02:15.061984 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:15.063559 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:15.063960 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:15.063961 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:15.063962 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:15.064043 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:15.064258 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:15.064260 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:15.064260 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:15.064411 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:15.064529 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:15.064673 (+ 144us) server_negotiation.cc:299] Negotiation successful
0926 04:02:15.064738 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 04:02:15.068342 18460 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37260:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 04:02:15.070220 18460 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 04:02:15.077096 18931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:15.072620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34447 (local address 127.0.0.1:47732)
0926 04:02:15.072790 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:15.072837 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 04:02:15.072909 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:02:15.073355 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:02:15.073356 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:02:15.073357 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:02:15.073393 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:15.073394 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:15.074744 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:15.074744 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:15.075862 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:15.075864 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:15.076772 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:15.076773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:15.076858 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:15.076874 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 04:02:15.076955 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 04:02:15.077011 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":104,"thread_start_us":84,"threads_started":1}
I20260926 04:02:15.077117 18935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:15.072989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47732 (local address 127.0.98.3:34447)
0926 04:02:15.073187 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:15.073191 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:15.073199 (+ 8us) server_negotiation.cc:407] Connection header received
0926 04:02:15.073236 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:15.073238 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:15.073253 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:15.073326 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:02:15.073478 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:15.074637 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:15.075942 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:15.076686 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:15.076763 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:15.076823 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:15.076980 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 04:02:15.077021 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":82,"threads_started":1}
I20260926 04:02:15.077477 18933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:15.072843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39061 (local address 127.0.0.1:48344)
0926 04:02:15.073435 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:15.073445 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 04:02:15.073488 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:02:15.073757 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:02:15.073758 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:02:15.073760 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:02:15.073815 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:15.073816 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:15.074679 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:15.074680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:15.076053 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:15.076054 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:15.077091 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:15.077093 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:15.077305 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:15.077311 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 04:02:15.077364 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 04:02:15.077411 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":573,"mutex_wait_us":49,"thread_start_us":78,"threads_started":1}
I20260926 04:02:15.077808 18934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:15.072963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48344 (local address 127.0.98.1:39061)
0926 04:02:15.073615 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:15.073619 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:15.073628 (+ 9us) server_negotiation.cc:407] Connection header received
0926 04:02:15.073667 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:15.073669 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:15.073686 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:15.073762 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:02:15.073891 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:15.074567 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:15.076142 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:15.076916 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:15.076998 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:15.077056 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:15.077667 (+ 611us) server_negotiation.cc:299] Negotiation successful
0926 04:02:15.077730 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":65,"threads_started":1}
I20260926 04:02:15.077845 18930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:15.072619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36689 (local address 127.0.0.1:39644)
0926 04:02:15.072989 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:15.073000 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 04:02:15.073044 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:02:15.073239 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:02:15.073241 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:02:15.073244 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:02:15.073331 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:15.073333 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:15.074886 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:15.074887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:15.076050 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:15.076053 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:15.077669 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:15.077670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:15.077736 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:15.077740 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 04:02:15.077771 (+ 31us) client_negotiation.cc:240] Negotiation successful
0926 04:02:15.077799 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"mutex_wait_us":23,"thread_start_us":107,"threads_started":1}
I20260926 04:02:15.077872 18853 tablet_service.cc:1467] Processing CreateTablet for tablet 2957fec8f688464e87f9171ed6d249e2 (DEFAULT_TABLE table=test-table [id=7a79f91fc18f41d99775e1c82f946fe6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:02:15.078248 18853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2957fec8f688464e87f9171ed6d249e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:02:15.078251 18932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:15.072768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39644 (local address 127.0.98.2:36689)
0926 04:02:15.073058 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:15.073063 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:15.073073 (+ 10us) server_negotiation.cc:407] Connection header received
0926 04:02:15.073112 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:15.073114 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:15.073131 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:15.073218 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:02:15.073435 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:15.074638 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:15.076742 (+ 2104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:15.077559 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:15.078049 (+ 490us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:15.078114 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:15.078140 (+ 26us) server_negotiation.cc:299] Negotiation successful
0926 04:02:15.078179 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":78,"threads_started":1}
I20260926 04:02:15.078568 18585 tablet_service.cc:1467] Processing CreateTablet for tablet 2957fec8f688464e87f9171ed6d249e2 (DEFAULT_TABLE table=test-table [id=7a79f91fc18f41d99775e1c82f946fe6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:02:15.078922 18585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2957fec8f688464e87f9171ed6d249e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:02:15.078867 18719 tablet_service.cc:1467] Processing CreateTablet for tablet 2957fec8f688464e87f9171ed6d249e2 (DEFAULT_TABLE table=test-table [id=7a79f91fc18f41d99775e1c82f946fe6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:02:15.079156 18719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2957fec8f688464e87f9171ed6d249e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:02:15.081595 18936 tablet_bootstrap.cc:492] T 2957fec8f688464e87f9171ed6d249e2 P 1f853c762d8a4074acf9bd2bb54f1c21: Bootstrap starting.
I20260926 04:02:15.082266 18936 tablet_bootstrap.cc:654] T 2957fec8f688464e87f9171ed6d249e2 P 1f853c762d8a4074acf9bd2bb54f1c21: Neither blocks nor log segments found. Creating new log.
I20260926 04:02:15.082291 18938 tablet_bootstrap.cc:492] T 2957fec8f688464e87f9171ed6d249e2 P c10aa3ad615f49219ffdcfbda80e603d: Bootstrap starting.
I20260926 04:02:15.082861 18936 log.cc:824] T 2957fec8f688464e87f9171ed6d249e2 P 1f853c762d8a4074acf9bd2bb54f1c21: Log is configured to *not* fsync() on all Append() calls
I20260926 04:02:15.082954 18938 tablet_bootstrap.cc:654] T 2957fec8f688464e87f9171ed6d249e2 P c10aa3ad615f49219ffdcfbda80e603d: Neither blocks nor log segments found. Creating new log.
I20260926 04:02:15.083256 18937 tablet_bootstrap.cc:492] T 2957fec8f688464e87f9171ed6d249e2 P ebdda04d615b458eaa36d7022778dc22: Bootstrap starting.
I20260926 04:02:15.083498 18938 log.cc:824] T 2957fec8f688464e87f9171ed6d249e2 P c10aa3ad615f49219ffdcfbda80e603d: Log is configured to *not* fsync() on all Append() calls
I20260926 04:02:15.083673 18936 tablet_bootstrap.cc:492] T 2957fec8f688464e87f9171ed6d249e2 P 1f853c762d8a4074acf9bd2bb54f1c21: No bootstrap required, opened a new log
I20260926 04:02:15.083761 18936 ts_tablet_manager.cc:1397] T 2957fec8f688464e87f9171ed6d249e2 P 1f853c762d8a4074acf9bd2bb54f1c21: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 04:02:15.083941 18937 tablet_bootstrap.cc:654] T 2957fec8f688464e87f9171ed6d249e2 P ebdda04d615b458eaa36d7022778dc22: Neither blocks nor log segments found. Creating new log.
I20260926 04:02:15.084398 18937 log.cc:824] T 2957fec8f688464e87f9171ed6d249e2 P ebdda04d615b458eaa36d7022778dc22: Log is configured to *not* fsync() on all Append() calls
I20260926 04:02:15.085168 18938 tablet_bootstrap.cc:492] T 2957fec8f688464e87f9171ed6d249e2 P c10aa3ad615f49219ffdcfbda80e603d: No bootstrap required, opened a new log
I20260926 04:02:15.085178 18937 tablet_bootstrap.cc:492] T 2957fec8f688464e87f9171ed6d249e2 P ebdda04d615b458eaa36d7022778dc22: No bootstrap required, opened a new log
I20260926 04:02:15.085242 18937 ts_tablet_manager.cc:1397] T 2957fec8f688464e87f9171ed6d249e2 P ebdda04d615b458eaa36d7022778dc22: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 04:02:15.085242 18938 ts_tablet_manager.cc:1397] T 2957fec8f688464e87f9171ed6d249e2 P c10aa3ad615f49219ffdcfbda80e603d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 04:02:15.085603 18936 raft_consensus.cc:348] T 2957fec8f688464e87f9171ed6d249e2 P 1f853c762d8a4074acf9bd2bb54f1c21 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c10aa3ad615f49219ffdcfbda80e603d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36689 } } peers { permanent_uuid: "ebdda04d615b458eaa36d7022778dc22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34447 } } peers { permanent_uuid: "1f853c762d8a4074acf9bd2bb54f1c21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39061 } }
I20260926 04:02:15.085769 18936 raft_consensus.cc:374] T 2957fec8f688464e87f9171ed6d249e2 P 1f853c762d8a4074acf9bd2bb54f1c21 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:02:15.085812 18936 raft_consensus.cc:729] T 2957fec8f688464e87f9171ed6d249e2 P 1f853c762d8a4074acf9bd2bb54f1c21 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f853c762d8a4074acf9bd2bb54f1c21, State: Initialized, Role: FOLLOWER
I20260926 04:02:15.085924 18936 consensus_queue.cc:260] T 2957fec8f688464e87f9171ed6d249e2 P 1f853c762d8a4074acf9bd2bb54f1c21 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c10aa3ad615f49219ffdcfbda80e603d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36689 } } peers { permanent_uuid: "ebdda04d615b458eaa36d7022778dc22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34447 } } peers { permanent_uuid: "1f853c762d8a4074acf9bd2bb54f1c21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39061 } }
I20260926 04:02:15.086212 18651 heartbeater.cc:503] Master 127.0.98.62:41111 was elected leader, sending a full tablet report...
I20260926 04:02:15.086474 18936 ts_tablet_manager.cc:1428] T 2957fec8f688464e87f9171ed6d249e2 P 1f853c762d8a4074acf9bd2bb54f1c21: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260926 04:02:15.087154 18938 raft_consensus.cc:348] T 2957fec8f688464e87f9171ed6d249e2 P c10aa3ad615f49219ffdcfbda80e603d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c10aa3ad615f49219ffdcfbda80e603d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36689 } } peers { permanent_uuid: "ebdda04d615b458eaa36d7022778dc22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34447 } } peers { permanent_uuid: "1f853c762d8a4074acf9bd2bb54f1c21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39061 } }
I20260926 04:02:15.087332 18938 raft_consensus.cc:374] T 2957fec8f688464e87f9171ed6d249e2 P c10aa3ad615f49219ffdcfbda80e603d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:02:15.087390 18938 raft_consensus.cc:729] T 2957fec8f688464e87f9171ed6d249e2 P c10aa3ad615f49219ffdcfbda80e603d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c10aa3ad615f49219ffdcfbda80e603d, State: Initialized, Role: FOLLOWER
I20260926 04:02:15.087512 18938 consensus_queue.cc:260] T 2957fec8f688464e87f9171ed6d249e2 P c10aa3ad615f49219ffdcfbda80e603d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c10aa3ad615f49219ffdcfbda80e603d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36689 } } peers { permanent_uuid: "ebdda04d615b458eaa36d7022778dc22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34447 } } peers { permanent_uuid: "1f853c762d8a4074acf9bd2bb54f1c21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39061 } }
I20260926 04:02:15.087774 18938 ts_tablet_manager.cc:1428] T 2957fec8f688464e87f9171ed6d249e2 P c10aa3ad615f49219ffdcfbda80e603d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 04:02:15.087796 18785 heartbeater.cc:503] Master 127.0.98.62:41111 was elected leader, sending a full tablet report...
I20260926 04:02:15.087705 18937 raft_consensus.cc:348] T 2957fec8f688464e87f9171ed6d249e2 P ebdda04d615b458eaa36d7022778dc22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c10aa3ad615f49219ffdcfbda80e603d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36689 } } peers { permanent_uuid: "ebdda04d615b458eaa36d7022778dc22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34447 } } peers { permanent_uuid: "1f853c762d8a4074acf9bd2bb54f1c21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39061 } }
I20260926 04:02:15.087864 18937 raft_consensus.cc:374] T 2957fec8f688464e87f9171ed6d249e2 P ebdda04d615b458eaa36d7022778dc22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:02:15.087915 18937 raft_consensus.cc:729] T 2957fec8f688464e87f9171ed6d249e2 P ebdda04d615b458eaa36d7022778dc22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebdda04d615b458eaa36d7022778dc22, State: Initialized, Role: FOLLOWER
I20260926 04:02:15.088032 18937 consensus_queue.cc:260] T 2957fec8f688464e87f9171ed6d249e2 P ebdda04d615b458eaa36d7022778dc22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c10aa3ad615f49219ffdcfbda80e603d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36689 } } peers { permanent_uuid: "ebdda04d615b458eaa36d7022778dc22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34447 } } peers { permanent_uuid: "1f853c762d8a4074acf9bd2bb54f1c21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39061 } }
I20260926 04:02:15.088289 18937 ts_tablet_manager.cc:1428] T 2957fec8f688464e87f9171ed6d249e2 P ebdda04d615b458eaa36d7022778dc22: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 04:02:15.088299 18919 heartbeater.cc:503] Master 127.0.98.62:41111 was elected leader, sending a full tablet report...
W20260926 04:02:15.136464 18786 tablet.cc:2369] T 2957fec8f688464e87f9171ed6d249e2 P c10aa3ad615f49219ffdcfbda80e603d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 04:02:15.213486 18652 tablet.cc:2369] T 2957fec8f688464e87f9171ed6d249e2 P 1f853c762d8a4074acf9bd2bb54f1c21: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 04:02:15.268744 18942 raft_consensus.cc:482] T 2957fec8f688464e87f9171ed6d249e2 P 1f853c762d8a4074acf9bd2bb54f1c21 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 04:02:15.268883 18942 raft_consensus.cc:504] T 2957fec8f688464e87f9171ed6d249e2 P 1f853c762d8a4074acf9bd2bb54f1c21 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c10aa3ad615f49219ffdcfbda80e603d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36689 } } peers { permanent_uuid: "ebdda04d615b458eaa36d7022778dc22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34447 } } peers { permanent_uuid: "1f853c762d8a4074acf9bd2bb54f1c21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39061 } }
I20260926 04:02:15.269232 18942 leader_election.cc:288] T 2957fec8f688464e87f9171ed6d249e2 P 1f853c762d8a4074acf9bd2bb54f1c21 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c10aa3ad615f49219ffdcfbda80e603d (127.0.98.2:36689), ebdda04d615b458eaa36d7022778dc22 (127.0.98.3:34447)
I20260926 04:02:15.272626 18932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:15.269533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47595 (local address 127.0.98.2:36689)
0926 04:02:15.269594 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:15.269596 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:15.269765 (+ 169us) server_negotiation.cc:407] Connection header received
0926 04:02:15.269849 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:15.269850 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:15.269869 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:15.269932 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:02:15.270126 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:15.270592 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:15.271867 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:15.272257 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:15.272350 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:15.272361 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:15.272513 (+ 152us) server_negotiation.cc:299] Negotiation successful
0926 04:02:15.272562 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 04:02:15.272639 18946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:15.269435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36689 (local address 127.0.98.1:47595)
0926 04:02:15.269690 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:15.269699 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 04:02:15.269779 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:02:15.269936 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:02:15.269937 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:02:15.269939 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:02:15.270020 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:15.270021 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:15.270684 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:15.270685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:15.271754 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:15.271757 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:15.272352 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:15.272353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:15.272432 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:15.272435 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 04:02:15.272507 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 04:02:15.272549 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":94,"threads_started":1}
I20260926 04:02:15.272755 18935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:15.269537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49077 (local address 127.0.98.3:34447)
0926 04:02:15.269592 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:15.269595 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:15.269694 (+ 99us) server_negotiation.cc:407] Connection header received
0926 04:02:15.269783 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:15.269784 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:15.269794 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:15.269868 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:02:15.270060 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:15.270531 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:15.271951 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:15.272329 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:15.272417 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:15.272427 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:15.272596 (+ 169us) server_negotiation.cc:299] Negotiation successful
0926 04:02:15.272660 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 04:02:15.272991 18945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:15.269429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34447 (local address 127.0.98.1:49077)
0926 04:02:15.269601 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:15.269612 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 04:02:15.269708 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:02:15.269875 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:02:15.269876 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:02:15.269878 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:02:15.269960 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:15.269962 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:15.270619 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:15.270620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:15.271649 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:15.271652 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:15.272420 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:15.272421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:15.272507 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:15.272511 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 04:02:15.272582 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 04:02:15.272624 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":94,"threads_started":1}
I20260926 04:02:15.273185 18739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2957fec8f688464e87f9171ed6d249e2" candidate_uuid: "1f853c762d8a4074acf9bd2bb54f1c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c10aa3ad615f49219ffdcfbda80e603d" is_pre_election: true
I20260926 04:02:15.273360 18739 raft_consensus.cc:2454] T 2957fec8f688464e87f9171ed6d249e2 P c10aa3ad615f49219ffdcfbda80e603d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f853c762d8a4074acf9bd2bb54f1c21 in term 0.
I20260926 04:02:15.273363 18873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2957fec8f688464e87f9171ed6d249e2" candidate_uuid: "1f853c762d8a4074acf9bd2bb54f1c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebdda04d615b458eaa36d7022778dc22" is_pre_election: true
I20260926 04:02:15.273520 18873 raft_consensus.cc:2454] T 2957fec8f688464e87f9171ed6d249e2 P ebdda04d615b458eaa36d7022778dc22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f853c762d8a4074acf9bd2bb54f1c21 in term 0.
I20260926 04:02:15.273615 18538 leader_election.cc:302] T 2957fec8f688464e87f9171ed6d249e2 P 1f853c762d8a4074acf9bd2bb54f1c21 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f853c762d8a4074acf9bd2bb54f1c21, c10aa3ad615f49219ffdcfbda80e603d; no voters:
I20260926 04:02:15.273802 18942 raft_consensus.cc:2781] T 2957fec8f688464e87f9171ed6d249e2 P 1f853c762d8a4074acf9bd2bb54f1c21 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 04:02:15.273845 18942 raft_consensus.cc:482] T 2957fec8f688464e87f9171ed6d249e2 P 1f853c762d8a4074acf9bd2bb54f1c21 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 04:02:15.273861 18942 raft_consensus.cc:3037] T 2957fec8f688464e87f9171ed6d249e2 P 1f853c762d8a4074acf9bd2bb54f1c21 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:02:15.274576 18942 raft_consensus.cc:504] T 2957fec8f688464e87f9171ed6d249e2 P 1f853c762d8a4074acf9bd2bb54f1c21 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c10aa3ad615f49219ffdcfbda80e603d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36689 } } peers { permanent_uuid: "ebdda04d615b458eaa36d7022778dc22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34447 } } peers { permanent_uuid: "1f853c762d8a4074acf9bd2bb54f1c21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39061 } }
I20260926 04:02:15.274722 18942 leader_election.cc:288] T 2957fec8f688464e87f9171ed6d249e2 P 1f853c762d8a4074acf9bd2bb54f1c21 [CANDIDATE]: Term 1 election: Requested vote from peers c10aa3ad615f49219ffdcfbda80e603d (127.0.98.2:36689), ebdda04d615b458eaa36d7022778dc22 (127.0.98.3:34447)
I20260926 04:02:15.274883 18873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2957fec8f688464e87f9171ed6d249e2" candidate_uuid: "1f853c762d8a4074acf9bd2bb54f1c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebdda04d615b458eaa36d7022778dc22"
I20260926 04:02:15.274976 18739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2957fec8f688464e87f9171ed6d249e2" candidate_uuid: "1f853c762d8a4074acf9bd2bb54f1c21" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c10aa3ad615f49219ffdcfbda80e603d"
I20260926 04:02:15.274991 18873 raft_consensus.cc:3037] T 2957fec8f688464e87f9171ed6d249e2 P ebdda04d615b458eaa36d7022778dc22 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:02:15.275058 18739 raft_consensus.cc:3037] T 2957fec8f688464e87f9171ed6d249e2 P c10aa3ad615f49219ffdcfbda80e603d [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:02:15.275887 18739 raft_consensus.cc:2454] T 2957fec8f688464e87f9171ed6d249e2 P c10aa3ad615f49219ffdcfbda80e603d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f853c762d8a4074acf9bd2bb54f1c21 in term 1.
I20260926 04:02:15.275887 18873 raft_consensus.cc:2454] T 2957fec8f688464e87f9171ed6d249e2 P ebdda04d615b458eaa36d7022778dc22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f853c762d8a4074acf9bd2bb54f1c21 in term 1.
I20260926 04:02:15.276073 18539 leader_election.cc:302] T 2957fec8f688464e87f9171ed6d249e2 P 1f853c762d8a4074acf9bd2bb54f1c21 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f853c762d8a4074acf9bd2bb54f1c21, ebdda04d615b458eaa36d7022778dc22; no voters:
I20260926 04:02:15.276181 18942 raft_consensus.cc:2781] T 2957fec8f688464e87f9171ed6d249e2 P 1f853c762d8a4074acf9bd2bb54f1c21 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:02:15.276355 18942 raft_consensus.cc:686] T 2957fec8f688464e87f9171ed6d249e2 P 1f853c762d8a4074acf9bd2bb54f1c21 [term 1 LEADER]: Becoming Leader. State: Replica: 1f853c762d8a4074acf9bd2bb54f1c21, State: Running, Role: LEADER
I20260926 04:02:15.276460 18942 consensus_queue.cc:237] T 2957fec8f688464e87f9171ed6d249e2 P 1f853c762d8a4074acf9bd2bb54f1c21 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c10aa3ad615f49219ffdcfbda80e603d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36689 } } peers { permanent_uuid: "ebdda04d615b458eaa36d7022778dc22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34447 } } peers { permanent_uuid: "1f853c762d8a4074acf9bd2bb54f1c21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39061 } }
I20260926 04:02:15.277314 18459 catalog_manager.cc:5495] T 2957fec8f688464e87f9171ed6d249e2 P 1f853c762d8a4074acf9bd2bb54f1c21 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f853c762d8a4074acf9bd2bb54f1c21 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "1f853c762d8a4074acf9bd2bb54f1c21" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c10aa3ad615f49219ffdcfbda80e603d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ebdda04d615b458eaa36d7022778dc22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34447 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f853c762d8a4074acf9bd2bb54f1c21" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39061 } health_report { overall_health: HEALTHY } } }
W20260926 04:02:15.290038 18920 tablet.cc:2369] T 2957fec8f688464e87f9171ed6d249e2 P ebdda04d615b458eaa36d7022778dc22: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 04:02:15.303956 18934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:15.301735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48356 (local address 127.0.98.1:39061)
0926 04:02:15.301799 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:15.301801 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:15.301809 (+ 8us) server_negotiation.cc:407] Connection header received
0926 04:02:15.301876 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:15.301876 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:15.301895 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:15.301947 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 04:02:15.302086 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:15.302556 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:15.303254 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:15.303432 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:15.303481 (+ 49us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:15.303632 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 04:02:15.303713 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 04:02:15.303783 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:15.303830 (+ 47us) server_negotiation.cc:299] Negotiation successful
0926 04:02:15.303840 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 04:02:15.305823 18873 raft_consensus.cc:1261] T 2957fec8f688464e87f9171ed6d249e2 P ebdda04d615b458eaa36d7022778dc22 [term 1 FOLLOWER]: Refusing update from remote peer 1f853c762d8a4074acf9bd2bb54f1c21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:02:15.305835 18739 raft_consensus.cc:1261] T 2957fec8f688464e87f9171ed6d249e2 P c10aa3ad615f49219ffdcfbda80e603d [term 1 FOLLOWER]: Refusing update from remote peer 1f853c762d8a4074acf9bd2bb54f1c21: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:02:15.306226 18947 consensus_queue.cc:1035] T 2957fec8f688464e87f9171ed6d249e2 P 1f853c762d8a4074acf9bd2bb54f1c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebdda04d615b458eaa36d7022778dc22" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34447 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:02:15.306370 18942 consensus_queue.cc:1035] T 2957fec8f688464e87f9171ed6d249e2 P 1f853c762d8a4074acf9bd2bb54f1c21 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c10aa3ad615f49219ffdcfbda80e603d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:02:15.308475 18950 mvcc.cc:205] Tried to move back new op lower bound from 7333459293409890304 to 7333459293293207552. Current Snapshot: MvccSnapshot[applied={T|T < 7333459293409890304}]
I20260926 04:02:15.308751 18951 mvcc.cc:205] Tried to move back new op lower bound from 7333459293409890304 to 7333459293293207552. Current Snapshot: MvccSnapshot[applied={T|T < 7333459293409890304}]
I20260926 04:02:15.311324 18459 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37260:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 04:02:15.311414 18459 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37260:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 04:02:15.312103 18459 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8b90b5633d314eb69c8f6b00bb97ea67: Sending DeleteTablet for 3 replicas of tablet 2957fec8f688464e87f9171ed6d249e2
I20260926 04:02:15.312441 18585 tablet_service.cc:1514] Processing DeleteTablet for tablet 2957fec8f688464e87f9171ed6d249e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:02:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48344
I20260926 04:02:15.312454 18853 tablet_service.cc:1514] Processing DeleteTablet for tablet 2957fec8f688464e87f9171ed6d249e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:02:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47732
I20260926 04:02:15.312443 18719 tablet_service.cc:1514] Processing DeleteTablet for tablet 2957fec8f688464e87f9171ed6d249e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:02:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39644
I20260926 04:02:15.312723 18959 tablet_replica.cc:330] T 2957fec8f688464e87f9171ed6d249e2 P 1f853c762d8a4074acf9bd2bb54f1c21: stopping tablet replica
I20260926 04:02:15.312826 18961 tablet_replica.cc:330] T 2957fec8f688464e87f9171ed6d249e2 P ebdda04d615b458eaa36d7022778dc22: stopping tablet replica
I20260926 04:02:15.312831 18959 raft_consensus.cc:2229] T 2957fec8f688464e87f9171ed6d249e2 P 1f853c762d8a4074acf9bd2bb54f1c21 [term 1 LEADER]: Raft consensus shutting down.
I20260926 04:02:15.312906 18961 raft_consensus.cc:2229] T 2957fec8f688464e87f9171ed6d249e2 P ebdda04d615b458eaa36d7022778dc22 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 04:02:15.312948 18959 raft_consensus.cc:2258] T 2957fec8f688464e87f9171ed6d249e2 P 1f853c762d8a4074acf9bd2bb54f1c21 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:02:15.312992 18961 raft_consensus.cc:2258] T 2957fec8f688464e87f9171ed6d249e2 P ebdda04d615b458eaa36d7022778dc22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:02:15.313146 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 18522
I20260926 04:02:15.313140 18960 tablet_replica.cc:330] T 2957fec8f688464e87f9171ed6d249e2 P c10aa3ad615f49219ffdcfbda80e603d: stopping tablet replica
I20260926 04:02:15.313879 18960 raft_consensus.cc:2229] T 2957fec8f688464e87f9171ed6d249e2 P c10aa3ad615f49219ffdcfbda80e603d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 04:02:15.313980 18960 raft_consensus.cc:2258] T 2957fec8f688464e87f9171ed6d249e2 P c10aa3ad615f49219ffdcfbda80e603d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:02:15.314236 18961 ts_tablet_manager.cc:1905] T 2957fec8f688464e87f9171ed6d249e2 P ebdda04d615b458eaa36d7022778dc22: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:02:15.314967 18960 ts_tablet_manager.cc:1905] T 2957fec8f688464e87f9171ed6d249e2 P c10aa3ad615f49219ffdcfbda80e603d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:02:15.316113 18961 ts_tablet_manager.cc:1918] T 2957fec8f688464e87f9171ed6d249e2 P ebdda04d615b458eaa36d7022778dc22: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 04:02:15.316217 18961 log.cc:1196] T 2957fec8f688464e87f9171ed6d249e2 P ebdda04d615b458eaa36d7022778dc22: Deleting WAL directory at /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/minicluster-data/ts-2/wal/wals/2957fec8f688464e87f9171ed6d249e2
I20260926 04:02:15.316524 18961 ts_tablet_manager.cc:1939] T 2957fec8f688464e87f9171ed6d249e2 P ebdda04d615b458eaa36d7022778dc22: Deleting consensus metadata
I20260926 04:02:15.316847 18960 ts_tablet_manager.cc:1918] T 2957fec8f688464e87f9171ed6d249e2 P c10aa3ad615f49219ffdcfbda80e603d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 04:02:15.316915 18960 log.cc:1196] T 2957fec8f688464e87f9171ed6d249e2 P c10aa3ad615f49219ffdcfbda80e603d: Deleting WAL directory at /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790395095474643-392-0/minicluster-data/ts-1/wal/wals/2957fec8f688464e87f9171ed6d249e2
I20260926 04:02:15.316954 18445 catalog_manager.cc:4841] TS ebdda04d615b458eaa36d7022778dc22 (127.0.98.3:34447): tablet 2957fec8f688464e87f9171ed6d249e2 (table test-table [id=7a79f91fc18f41d99775e1c82f946fe6]) successfully deleted
I20260926 04:02:15.317174 18960 ts_tablet_manager.cc:1939] T 2957fec8f688464e87f9171ed6d249e2 P c10aa3ad615f49219ffdcfbda80e603d: Deleting consensus metadata
I20260926 04:02:15.317889 18444 catalog_manager.cc:4841] TS c10aa3ad615f49219ffdcfbda80e603d (127.0.98.2:36689): tablet 2957fec8f688464e87f9171ed6d249e2 (table test-table [id=7a79f91fc18f41d99775e1c82f946fe6]) successfully deleted
W20260926 04:02:15.319458 18445 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39061 (error 108)
I20260926 04:02:15.319998 18930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:15.319835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39061 (local address 127.0.0.1:48368)
0926 04:02:15.319892 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:15.319959 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 04:02:15.320016 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 18656
W20260926 04:02:15.321199 18445 catalog_manager.cc:4584] TS 1f853c762d8a4074acf9bd2bb54f1c21 (127.0.98.1:39061): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2957fec8f688464e87f9171ed6d249e2: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39061: connect: Connection refused (error 111)
I20260926 04:02:15.324460 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 18790
I20260926 04:02:15.328992 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 18428
2026-09-26T04:02:15Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2114 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-26T04:02:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:02:15Z Disabled control of system clock
I20260926 04:02:15.355917 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395095474643-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395095474643-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395095474643-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39963
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:44601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39963
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 04:02:15.435225 18966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:02:15.435420 18966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:02:15.435439 18966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:02:15.436475 18966 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:02:15.436507 18966 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:02:15.436519 18966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:02:15.436532 18966 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:02:15.436544 18966 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:02:15.437793 18966 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395095474643-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395095474643-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39963
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18966
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395095474643-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:02:15.437991 18966 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:02:15.438221 18966 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:02:15.442939 18974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:02:15.443105 18966 server_base.cc:1034] running on GCE node
W20260926 04:02:15.442929 18972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:02:15.442929 18971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:02:15.443418 18966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:02:15.443696 18966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:02:15.444852 18966 hybrid_clock.cc:648] HybridClock initialized: now 1790395335444829 us; error 41 us; skew 500 ppm
I20260926 04:02:15.450127 18966 webserver.cc:460] Webserver started at http://127.0.98.62:45735/ using document root <none> and password file <none>
I20260926 04:02:15.450405 18966 fs_manager.cc:359] Metadata directory not provided
I20260926 04:02:15.450475 18966 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:02:15.450608 18966 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:02:15.452138 18966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395095474643-392-0/minicluster-data/master-0/data/instance:
uuid: "4d1787dcec644fbf91d03f66edd3e3d8"
format_stamp: "Formatted at 2026-09-26 04:02:15 on dist-test-slave-nhpw"
server_key: "304e84944234d3bd312b2fb000c53b92"
server_key_iv: "43a52ad0c2ac1f1164ae731579ba5ae4"
server_key_version: "encryptionkey@0"
I20260926 04:02:15.452581 18966 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395095474643-392-0/minicluster-data/master-0/wal/instance:
uuid: "4d1787dcec644fbf91d03f66edd3e3d8"
format_stamp: "Formatted at 2026-09-26 04:02:15 on dist-test-slave-nhpw"
server_key: "304e84944234d3bd312b2fb000c53b92"
server_key_iv: "43a52ad0c2ac1f1164ae731579ba5ae4"
server_key_version: "encryptionkey@0"
I20260926 04:02:15.454429 18966 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260926 04:02:15.455343 18980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:15.455523 18966 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:02:15.455616 18966 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395095474643-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395095474643-392-0/minicluster-data/master-0/wal
uuid: "4d1787dcec644fbf91d03f66edd3e3d8"
format_stamp: "Formatted at 2026-09-26 04:02:15 on dist-test-slave-nhpw"
server_key: "304e84944234d3bd312b2fb000c53b92"
server_key_iv: "43a52ad0c2ac1f1164ae731579ba5ae4"
server_key_version: "encryptionkey@0"
I20260926 04:02:15.455699 18966 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395095474643-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395095474643-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395095474643-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:02:15.468149 18966 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:02:15.468406 18966 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:02:15.468508 18966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:02:15.472361 18966 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39963
I20260926 04:02:15.472396 19032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39963 every 8 connection(s)
I20260926 04:02:15.472772 18966 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
I20260926 04:02:15.473716 19033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:02:15.475965 19033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d1787dcec644fbf91d03f66edd3e3d8: Bootstrap starting.
I20260926 04:02:15.476548 19033 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d1787dcec644fbf91d03f66edd3e3d8: Neither blocks nor log segments found. Creating new log.
I20260926 04:02:15.476863 19033 log.cc:824] T 00000000000000000000000000000000 P 4d1787dcec644fbf91d03f66edd3e3d8: Log is configured to *not* fsync() on all Append() calls
I20260926 04:02:15.477550 19033 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d1787dcec644fbf91d03f66edd3e3d8: No bootstrap required, opened a new log
I20260926 04:02:15.479137 19033 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d1787dcec644fbf91d03f66edd3e3d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d1787dcec644fbf91d03f66edd3e3d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39963 } }
I20260926 04:02:15.479301 19033 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d1787dcec644fbf91d03f66edd3e3d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:02:15.479349 19033 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d1787dcec644fbf91d03f66edd3e3d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d1787dcec644fbf91d03f66edd3e3d8, State: Initialized, Role: FOLLOWER
I20260926 04:02:15.479542 19033 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d1787dcec644fbf91d03f66edd3e3d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d1787dcec644fbf91d03f66edd3e3d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39963 } }
I20260926 04:02:15.479645 19033 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d1787dcec644fbf91d03f66edd3e3d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:02:15.479688 19033 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d1787dcec644fbf91d03f66edd3e3d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:02:15.479736 19033 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d1787dcec644fbf91d03f66edd3e3d8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:02:15.480022 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 18966
I20260926 04:02:15.480106 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395095474643-392-0/minicluster-data/master-0/wal/instance
I20260926 04:02:15.480298 392 external_mini_cluster.cc:1428] Setting key 1a64aebe681ef9971b01059a2aef11b8
I20260926 04:02:15.480368 19033 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d1787dcec644fbf91d03f66edd3e3d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d1787dcec644fbf91d03f66edd3e3d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39963 } }
I20260926 04:02:15.480484 19033 leader_election.cc:302] T 00000000000000000000000000000000 P 4d1787dcec644fbf91d03f66edd3e3d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d1787dcec644fbf91d03f66edd3e3d8; no voters:
I20260926 04:02:15.480692 19033 leader_election.cc:288] T 00000000000000000000000000000000 P 4d1787dcec644fbf91d03f66edd3e3d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 04:02:15.480723 19036 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d1787dcec644fbf91d03f66edd3e3d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:02:15.480960 19033 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d1787dcec644fbf91d03f66edd3e3d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:02:15.481112 19036 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d1787dcec644fbf91d03f66edd3e3d8 [term 1 LEADER]: Becoming Leader. State: Replica: 4d1787dcec644fbf91d03f66edd3e3d8, State: Running, Role: LEADER
I20260926 04:02:15.481218 19036 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d1787dcec644fbf91d03f66edd3e3d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d1787dcec644fbf91d03f66edd3e3d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39963 } }
I20260926 04:02:15.481528 19036 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d1787dcec644fbf91d03f66edd3e3d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d1787dcec644fbf91d03f66edd3e3d8. Latest consensus state: current_term: 1 leader_uuid: "4d1787dcec644fbf91d03f66edd3e3d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d1787dcec644fbf91d03f66edd3e3d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39963 } } }
I20260926 04:02:15.481612 19036 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d1787dcec644fbf91d03f66edd3e3d8 [sys.catalog]: This master's current role is: LEADER
I20260926 04:02:15.481696 19038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d1787dcec644fbf91d03f66edd3e3d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d1787dcec644fbf91d03f66edd3e3d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d1787dcec644fbf91d03f66edd3e3d8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39963 } } }
I20260926 04:02:15.481763 19038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d1787dcec644fbf91d03f66edd3e3d8 [sys.catalog]: This master's current role is: LEADER
I20260926 04:02:15.482465 19046 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:02:15.482915 19046 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:02:15.484366 19046 catalog_manager.cc:1347] Generated new cluster ID: f900161045e24824853e7dee29cd168d
I20260926 04:02:15.484424 19046 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 04:02:15.487022 19039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:15.480858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41794 (local address 127.0.98.62:39963)
0926 04:02:15.481068 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:15.481078 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:15.481114 (+ 36us) server_negotiation.cc:407] Connection header received
0926 04:02:15.481232 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:15.481239 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:15.481308 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:15.481470 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:15.481916 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:15.483319 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:15.484425 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:15.485076 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:15.485199 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:15.486117 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:15.486120 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:15.486121 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:02:15.486322 (+ 201us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:15.486368 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:02:15.486383 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:15.486445 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:15.486702 (+ 257us) server_negotiation.cc:299] Negotiation successful
0926 04:02:15.486782 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":82,"threads_started":1}
I20260926 04:02:15.515689 19046 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:02:15.516445 19046 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:02:15.527916 19046 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d1787dcec644fbf91d03f66edd3e3d8: Generated new TSK 0
I20260926 04:02:15.528127 19046 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 04:02:15.535687 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395095474643-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395095474643-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395095474643-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39963
--builtin_ntp_servers=127.0.98.20:44601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 04:02:15.615480 19057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:02:15.615731 19057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:02:15.615764 19057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:02:15.616926 19057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:02:15.616995 19057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:02:15.617034 19057 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 04:02:15.618397 19057 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395095474643-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395095474643-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19057
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395095474643-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:02:15.618676 19057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:02:15.618973 19057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:02:15.623956 19065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:02:15.623962 19062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:02:15.623951 19063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:02:15.624240 19057 server_base.cc:1034] running on GCE node
I20260926 04:02:15.624411 19057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:02:15.624593 19057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:02:15.625792 19057 hybrid_clock.cc:648] HybridClock initialized: now 1790395335625778 us; error 27 us; skew 500 ppm
I20260926 04:02:15.627106 19057 webserver.cc:460] Webserver started at http://127.0.98.1:34747/ using document root <none> and password file <none>
I20260926 04:02:15.627345 19057 fs_manager.cc:359] Metadata directory not provided
I20260926 04:02:15.627393 19057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:02:15.627561 19057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:02:15.628607 19057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395095474643-392-0/minicluster-data/ts-0/data/instance:
uuid: "02d556b947994df88dda40ff5e74c9bc"
format_stamp: "Formatted at 2026-09-26 04:02:15 on dist-test-slave-nhpw"
server_key: "ec6e19f22731fd757173652fd43e8eef"
server_key_iv: "f8914671a65f3478c95783e6d4ecea9a"
server_key_version: "encryptionkey@0"
I20260926 04:02:15.629010 19057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395095474643-392-0/minicluster-data/ts-0/wal/instance:
uuid: "02d556b947994df88dda40ff5e74c9bc"
format_stamp: "Formatted at 2026-09-26 04:02:15 on dist-test-slave-nhpw"
server_key: "ec6e19f22731fd757173652fd43e8eef"
server_key_iv: "f8914671a65f3478c95783e6d4ecea9a"
server_key_version: "encryptionkey@0"
I20260926 04:02:15.630267 19057 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:02:15.630939 19071 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:15.631120 19057 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 04:02:15.631220 19057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395095474643-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395095474643-392-0/minicluster-data/ts-0/wal
uuid: "02d556b947994df88dda40ff5e74c9bc"
format_stamp: "Formatted at 2026-09-26 04:02:15 on dist-test-slave-nhpw"
server_key: "ec6e19f22731fd757173652fd43e8eef"
server_key_iv: "f8914671a65f3478c95783e6d4ecea9a"
server_key_version: "encryptionkey@0"
I20260926 04:02:15.631291 19057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395095474643-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395095474643-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395095474643-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:02:15.641661 19057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:02:15.641949 19057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:02:15.642104 19057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:02:15.642318 19057 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:02:15.642647 19057 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:02:15.642701 19057 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:15.642733 19057 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:02:15.642760 19057 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:15.648289 19057 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43179
I20260926 04:02:15.648371 19184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43179 every 8 connection(s)
I20260926 04:02:15.648779 19057 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
I20260926 04:02:15.650561 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 19057
I20260926 04:02:15.650679 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395095474643-392-0/minicluster-data/ts-0/wal/instance
I20260926 04:02:15.650869 392 external_mini_cluster.cc:1428] Setting key c64433d80d1bd75f5b594f05fe14a4c5
I20260926 04:02:15.651849 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395095474643-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395095474643-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395095474643-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39963
--builtin_ntp_servers=127.0.98.20:44601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 04:02:15.654553 19039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:15.649539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41851 (local address 127.0.98.62:39963)
0926 04:02:15.649590 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:15.649592 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:15.649843 (+ 251us) server_negotiation.cc:407] Connection header received
0926 04:02:15.650329 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:15.650330 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:15.650340 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:15.650426 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:15.651258 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:15.651975 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:15.653083 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:15.653299 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:15.653378 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:15.654290 (+ 912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:15.654291 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:15.654292 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:02:15.654296 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:15.654311 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:02:15.654315 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:15.654361 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:15.654458 (+ 97us) server_negotiation.cc:299] Negotiation successful
0926 04:02:15.654486 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 04:02:15.654843 19187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:15.649489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39963 (local address 127.0.98.1:41851)
0926 04:02:15.649691 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:15.649733 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 04:02:15.650243 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:02:15.650503 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:02:15.650506 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:02:15.650514 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:02:15.651166 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:15.651174 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:15.652122 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:15.652123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:15.652985 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:15.652988 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:15.653387 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:15.653388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:15.653551 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:15.654135 (+ 584us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 04:02:15.654135 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:02:15.654172 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 04:02:15.654201 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:02:15.654379 (+ 178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:02:15.654384 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:02:15.654388 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 04:02:15.654461 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 04:02:15.654586 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":111,"threads_started":1}
I20260926 04:02:15.655301 19185 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39963
I20260926 04:02:15.655406 19185 heartbeater.cc:464] Registering TS with master...
I20260926 04:02:15.655582 19185 heartbeater.cc:511] Master 127.0.98.62:39963 requested a full tablet report, sending...
I20260926 04:02:15.656108 18997 ts_manager.cc:194] Registered new tserver with Master: 02d556b947994df88dda40ff5e74c9bc (127.0.98.1:43179)
I20260926 04:02:15.656800 18997 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:41851
W20260926 04:02:15.733168 19188 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:02:15.733424 19188 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:02:15.733461 19188 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:02:15.734727 19188 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:02:15.734786 19188 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:02:15.734827 19188 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 04:02:15.736147 19188 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395095474643-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395095474643-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19188
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395095474643-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:02:15.736402 19188 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:02:15.736716 19188 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:02:15.741760 19196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:02:15.741756 19193 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:02:15.741935 19188 server_base.cc:1034] running on GCE node
W20260926 04:02:15.741834 19194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:02:15.742245 19188 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:02:15.742460 19188 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:02:15.743613 19188 hybrid_clock.cc:648] HybridClock initialized: now 1790395335743597 us; error 28 us; skew 500 ppm
I20260926 04:02:15.745049 19188 webserver.cc:460] Webserver started at http://127.0.98.2:34179/ using document root <none> and password file <none>
I20260926 04:02:15.745302 19188 fs_manager.cc:359] Metadata directory not provided
I20260926 04:02:15.745350 19188 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:02:15.745530 19188 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:02:15.746603 19188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395095474643-392-0/minicluster-data/ts-1/data/instance:
uuid: "807d91621ef74150ba3d157c8596d3f5"
format_stamp: "Formatted at 2026-09-26 04:02:15 on dist-test-slave-nhpw"
server_key: "13ccd9bb7a448cdd73af7985acf8ef30"
server_key_iv: "e8f1666cdf07938231bea5319c36554c"
server_key_version: "encryptionkey@0"
I20260926 04:02:15.747036 19188 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395095474643-392-0/minicluster-data/ts-1/wal/instance:
uuid: "807d91621ef74150ba3d157c8596d3f5"
format_stamp: "Formatted at 2026-09-26 04:02:15 on dist-test-slave-nhpw"
server_key: "13ccd9bb7a448cdd73af7985acf8ef30"
server_key_iv: "e8f1666cdf07938231bea5319c36554c"
server_key_version: "encryptionkey@0"
I20260926 04:02:15.748315 19188 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 04:02:15.749089 19202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:15.749265 19188 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:15.749331 19188 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395095474643-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395095474643-392-0/minicluster-data/ts-1/wal
uuid: "807d91621ef74150ba3d157c8596d3f5"
format_stamp: "Formatted at 2026-09-26 04:02:15 on dist-test-slave-nhpw"
server_key: "13ccd9bb7a448cdd73af7985acf8ef30"
server_key_iv: "e8f1666cdf07938231bea5319c36554c"
server_key_version: "encryptionkey@0"
I20260926 04:02:15.749424 19188 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395095474643-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395095474643-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395095474643-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:02:15.759982 19188 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:02:15.760273 19188 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:02:15.760417 19188 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:02:15.760624 19188 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:02:15.761000 19188 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:02:15.761040 19188 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:15.761096 19188 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:02:15.761124 19188 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:15.766458 19188 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44747
I20260926 04:02:15.766527 19315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44747 every 8 connection(s)
I20260926 04:02:15.766852 19188 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
I20260926 04:02:15.767112 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 19188
I20260926 04:02:15.767218 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395095474643-392-0/minicluster-data/ts-1/wal/instance
I20260926 04:02:15.767469 392 external_mini_cluster.cc:1428] Setting key 39e6f391506ea6f7598553af86d2c51a
I20260926 04:02:15.768579 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395095474643-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395095474643-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395095474643-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39963
--builtin_ntp_servers=127.0.98.20:44601
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 04:02:15.773355 19039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:15.767810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39289 (local address 127.0.98.62:39963)
0926 04:02:15.767881 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:15.767883 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:15.768078 (+ 195us) server_negotiation.cc:407] Connection header received
0926 04:02:15.768803 (+ 725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:15.768804 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:15.768817 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:15.768847 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:15.769889 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:15.770368 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:15.771891 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:15.772075 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:15.772125 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:15.773118 (+ 993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:15.773119 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:15.773119 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:02:15.773123 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:15.773137 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:02:15.773141 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:15.773185 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:15.773271 (+ 86us) server_negotiation.cc:299] Negotiation successful
0926 04:02:15.773296 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 04:02:15.773672 19318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:15.767710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39963 (local address 127.0.98.2:39289)
0926 04:02:15.767936 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:15.767984 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 04:02:15.768676 (+ 692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:02:15.769095 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:02:15.769099 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:02:15.769109 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:02:15.769793 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:15.769800 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:15.770473 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:15.770474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:15.771802 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:15.771805 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:15.772163 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:15.772164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:15.772334 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:15.772945 (+ 611us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 04:02:15.772946 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:02:15.772988 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 04:02:15.773020 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:02:15.773188 (+ 168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:02:15.773192 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:02:15.773196 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 04:02:15.773268 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 04:02:15.773385 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":95,"threads_started":1}
I20260926 04:02:15.774205 19316 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39963
I20260926 04:02:15.774309 19316 heartbeater.cc:464] Registering TS with master...
I20260926 04:02:15.774523 19316 heartbeater.cc:511] Master 127.0.98.62:39963 requested a full tablet report, sending...
I20260926 04:02:15.774915 18997 ts_manager.cc:194] Registered new tserver with Master: 807d91621ef74150ba3d157c8596d3f5 (127.0.98.2:44747)
I20260926 04:02:15.775292 18997 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:39289
W20260926 04:02:15.852412 19319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:02:15.852628 19319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:02:15.852735 19319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:02:15.853941 19319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:02:15.853991 19319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:02:15.854018 19319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 04:02:15.855399 19319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44601
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395095474643-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395095474643-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19319
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395095474643-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:02:15.855646 19319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:02:15.855948 19319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:02:15.860948 19325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:02:15.861111 19324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:02:15.861025 19327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:02:15.861246 19319 server_base.cc:1034] running on GCE node
I20260926 04:02:15.861442 19319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:02:15.861652 19319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:02:15.862828 19319 hybrid_clock.cc:648] HybridClock initialized: now 1790395335862811 us; error 28 us; skew 500 ppm
I20260926 04:02:15.864198 19319 webserver.cc:460] Webserver started at http://127.0.98.3:43915/ using document root <none> and password file <none>
I20260926 04:02:15.864485 19319 fs_manager.cc:359] Metadata directory not provided
I20260926 04:02:15.864537 19319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:02:15.864750 19319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:02:15.865921 19319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395095474643-392-0/minicluster-data/ts-2/data/instance:
uuid: "9fb0406cbed74da2b933887dd8cbcac0"
format_stamp: "Formatted at 2026-09-26 04:02:15 on dist-test-slave-nhpw"
server_key: "5f526b7cbf94bfbebe733f241f25cb4c"
server_key_iv: "7ebc1992061ed7060316d40431d4bb40"
server_key_version: "encryptionkey@0"
I20260926 04:02:15.866312 19319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395095474643-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9fb0406cbed74da2b933887dd8cbcac0"
format_stamp: "Formatted at 2026-09-26 04:02:15 on dist-test-slave-nhpw"
server_key: "5f526b7cbf94bfbebe733f241f25cb4c"
server_key_iv: "7ebc1992061ed7060316d40431d4bb40"
server_key_version: "encryptionkey@0"
I20260926 04:02:15.867695 19319 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 04:02:15.868423 19333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:15.868619 19319 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 04:02:15.868767 19319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395095474643-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395095474643-392-0/minicluster-data/ts-2/wal
uuid: "9fb0406cbed74da2b933887dd8cbcac0"
format_stamp: "Formatted at 2026-09-26 04:02:15 on dist-test-slave-nhpw"
server_key: "5f526b7cbf94bfbebe733f241f25cb4c"
server_key_iv: "7ebc1992061ed7060316d40431d4bb40"
server_key_version: "encryptionkey@0"
I20260926 04:02:15.868851 19319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395095474643-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395095474643-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395095474643-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:02:15.902766 19319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:02:15.903119 19319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:02:15.903280 19319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:02:15.903537 19319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:02:15.903874 19319 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:02:15.903929 19319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:15.903966 19319 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:02:15.904001 19319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:15.909641 19319 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34235
I20260926 04:02:15.909706 19446 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34235 every 8 connection(s)
I20260926 04:02:15.910023 19319 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
I20260926 04:02:15.914309 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 19319
I20260926 04:02:15.914433 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395095474643-392-0/minicluster-data/ts-2/wal/instance
I20260926 04:02:15.914618 392 external_mini_cluster.cc:1428] Setting key 7578415695be95949459150e350fe166
I20260926 04:02:15.915174 19039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:15.910749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58171 (local address 127.0.98.62:39963)
0926 04:02:15.910839 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:15.910842 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:15.911050 (+ 208us) server_negotiation.cc:407] Connection header received
0926 04:02:15.911520 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:15.911521 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:15.911532 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:15.911592 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:15.912203 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:15.912716 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:15.913624 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:15.913807 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:15.913856 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:15.914731 (+ 875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:15.914732 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:15.914733 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:02:15.914738 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:15.914755 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:02:15.914778 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:15.914834 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:15.915005 (+ 171us) server_negotiation.cc:299] Negotiation successful
0926 04:02:15.915051 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 04:02:15.915346 19449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:15.910713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39963 (local address 127.0.98.3:58171)
0926 04:02:15.910918 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:15.910962 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 04:02:15.911437 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:02:15.911640 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:02:15.911644 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:02:15.911651 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:02:15.912132 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:15.912136 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:15.912786 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:15.912787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:15.913552 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:15.913554 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:15.913876 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:15.913877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:15.913982 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:15.914573 (+ 591us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 04:02:15.914574 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:02:15.914602 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 04:02:15.914624 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:02:15.914845 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:02:15.914850 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:02:15.914853 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 04:02:15.914993 (+ 140us) client_negotiation.cc:240] Negotiation successful
0926 04:02:15.915123 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":106,"threads_started":1}
I20260926 04:02:15.915845 19447 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39963
I20260926 04:02:15.915961 19447 heartbeater.cc:464] Registering TS with master...
I20260926 04:02:15.916144 19447 heartbeater.cc:511] Master 127.0.98.62:39963 requested a full tablet report, sending...
I20260926 04:02:15.916522 18997 ts_manager.cc:194] Registered new tserver with Master: 9fb0406cbed74da2b933887dd8cbcac0 (127.0.98.3:34235)
I20260926 04:02:15.916910 18997 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:58171
I20260926 04:02:15.926581 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 04:02:15.931176 19039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:15.928048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41810 (local address 127.0.98.62:39963)
0926 04:02:15.928109 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:15.928111 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:15.928196 (+ 85us) server_negotiation.cc:407] Connection header received
0926 04:02:15.928266 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:15.928267 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:15.928278 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:15.928324 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:15.928516 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:15.929068 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:15.929831 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:15.930034 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:15.930110 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:15.930862 (+ 752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:15.930864 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:15.930864 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:02:15.930869 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:15.930883 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:02:15.930887 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:15.931005 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:15.931084 (+ 79us) server_negotiation.cc:299] Negotiation successful
0926 04:02:15.931095 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 04:02:15.932732 18997 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41810:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 04:02:15.934309 18997 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 04:02:15.940703 19456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:15.936533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44747 (local address 127.0.0.1:52312)
0926 04:02:15.936731 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:15.936770 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 04:02:15.936899 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:02:15.937403 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:02:15.937404 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:02:15.937406 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:02:15.937440 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:15.937441 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:15.938723 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:15.938724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:15.939673 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:15.939675 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:15.940375 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:15.940376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:15.940468 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:15.940481 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 04:02:15.940555 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 04:02:15.940611 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":58,"threads_started":1}
I20260926 04:02:15.940718 19457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:15.936619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52312 (local address 127.0.98.2:44747)
0926 04:02:15.937027 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:15.937033 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:15.937043 (+ 10us) server_negotiation.cc:407] Connection header received
0926 04:02:15.937086 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:15.937087 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:15.937102 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:15.937210 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:02:15.937523 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:15.938624 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:15.939760 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:15.940283 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:15.940375 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:15.940464 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:15.940575 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 04:02:15.940613 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":124,"threads_started":1}
I20260926 04:02:15.940805 19459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:15.936769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43179 (local address 127.0.0.1:51372)
0926 04:02:15.936875 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:15.936885 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 04:02:15.936936 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:02:15.937305 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:02:15.937306 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:02:15.937308 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:02:15.937357 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:15.937358 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:15.938694 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:15.938695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:15.939738 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:15.939740 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:15.940586 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:15.940586 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:15.940679 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:15.940683 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 04:02:15.940711 (+ 28us) client_negotiation.cc:240] Negotiation successful
0926 04:02:15.940739 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"mutex_wait_us":10,"thread_start_us":60,"threads_started":1}
I20260926 04:02:15.940804 19455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:15.936550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34235 (local address 127.0.0.1:60424)
0926 04:02:15.936830 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:15.936841 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 04:02:15.936882 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:02:15.937235 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:02:15.937238 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:02:15.937242 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:02:15.937309 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:15.937311 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:15.938390 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:15.938392 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:15.939635 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:15.939638 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:15.940563 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:15.940565 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:15.940626 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:15.940651 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 04:02:15.940682 (+ 31us) client_negotiation.cc:240] Negotiation successful
0926 04:02:15.940725 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":116,"threads_started":1}
I20260926 04:02:15.941133 19460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:15.936961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51372 (local address 127.0.98.1:43179)
0926 04:02:15.937151 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:15.937155 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:15.937165 (+ 10us) server_negotiation.cc:407] Connection header received
0926 04:02:15.937204 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:15.937206 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:15.937221 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:15.937304 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:02:15.937445 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:15.938604 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:15.939823 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:15.940502 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:15.940890 (+ 388us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:15.940979 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:15.941006 (+ 27us) server_negotiation.cc:299] Negotiation successful
0926 04:02:15.941055 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":127,"threads_started":1}
I20260926 04:02:15.941169 19458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:15.936668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60424 (local address 127.0.98.3:34235)
0926 04:02:15.937036 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:15.937040 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:15.937049 (+ 9us) server_negotiation.cc:407] Connection header received
0926 04:02:15.937084 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:15.937086 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:15.937102 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:15.937211 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:02:15.937397 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:15.938304 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:15.939733 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:15.940443 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:15.940886 (+ 443us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:15.940978 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:15.941006 (+ 28us) server_negotiation.cc:299] Negotiation successful
0926 04:02:15.941044 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":332,"thread_start_us":78,"threads_started":1}
I20260926 04:02:15.941766 19250 tablet_service.cc:1467] Processing CreateTablet for tablet 84247e15954e4152a418c422f82b128f (DEFAULT_TABLE table=test-table [id=2a469236fa05442e9f862064ce441013]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:02:15.941766 19119 tablet_service.cc:1467] Processing CreateTablet for tablet 84247e15954e4152a418c422f82b128f (DEFAULT_TABLE table=test-table [id=2a469236fa05442e9f862064ce441013]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:02:15.941850 19381 tablet_service.cc:1467] Processing CreateTablet for tablet 84247e15954e4152a418c422f82b128f (DEFAULT_TABLE table=test-table [id=2a469236fa05442e9f862064ce441013]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:02:15.942126 19250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84247e15954e4152a418c422f82b128f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:02:15.942126 19119 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84247e15954e4152a418c422f82b128f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:02:15.942216 19381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84247e15954e4152a418c422f82b128f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:02:15.945034 19461 tablet_bootstrap.cc:492] T 84247e15954e4152a418c422f82b128f P 9fb0406cbed74da2b933887dd8cbcac0: Bootstrap starting.
I20260926 04:02:15.945380 19462 tablet_bootstrap.cc:492] T 84247e15954e4152a418c422f82b128f P 807d91621ef74150ba3d157c8596d3f5: Bootstrap starting.
I20260926 04:02:15.945780 19463 tablet_bootstrap.cc:492] T 84247e15954e4152a418c422f82b128f P 02d556b947994df88dda40ff5e74c9bc: Bootstrap starting.
I20260926 04:02:15.945884 19461 tablet_bootstrap.cc:654] T 84247e15954e4152a418c422f82b128f P 9fb0406cbed74da2b933887dd8cbcac0: Neither blocks nor log segments found. Creating new log.
I20260926 04:02:15.946035 19462 tablet_bootstrap.cc:654] T 84247e15954e4152a418c422f82b128f P 807d91621ef74150ba3d157c8596d3f5: Neither blocks nor log segments found. Creating new log.
I20260926 04:02:15.946307 19461 log.cc:824] T 84247e15954e4152a418c422f82b128f P 9fb0406cbed74da2b933887dd8cbcac0: Log is configured to *not* fsync() on all Append() calls
I20260926 04:02:15.946323 19462 log.cc:824] T 84247e15954e4152a418c422f82b128f P 807d91621ef74150ba3d157c8596d3f5: Log is configured to *not* fsync() on all Append() calls
I20260926 04:02:15.946391 19463 tablet_bootstrap.cc:654] T 84247e15954e4152a418c422f82b128f P 02d556b947994df88dda40ff5e74c9bc: Neither blocks nor log segments found. Creating new log.
I20260926 04:02:15.946710 19463 log.cc:824] T 84247e15954e4152a418c422f82b128f P 02d556b947994df88dda40ff5e74c9bc: Log is configured to *not* fsync() on all Append() calls
I20260926 04:02:15.947049 19461 tablet_bootstrap.cc:492] T 84247e15954e4152a418c422f82b128f P 9fb0406cbed74da2b933887dd8cbcac0: No bootstrap required, opened a new log
I20260926 04:02:15.947059 19462 tablet_bootstrap.cc:492] T 84247e15954e4152a418c422f82b128f P 807d91621ef74150ba3d157c8596d3f5: No bootstrap required, opened a new log
I20260926 04:02:15.947122 19462 ts_tablet_manager.cc:1397] T 84247e15954e4152a418c422f82b128f P 807d91621ef74150ba3d157c8596d3f5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 04:02:15.947122 19461 ts_tablet_manager.cc:1397] T 84247e15954e4152a418c422f82b128f P 9fb0406cbed74da2b933887dd8cbcac0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 04:02:15.947427 19463 tablet_bootstrap.cc:492] T 84247e15954e4152a418c422f82b128f P 02d556b947994df88dda40ff5e74c9bc: No bootstrap required, opened a new log
I20260926 04:02:15.947506 19463 ts_tablet_manager.cc:1397] T 84247e15954e4152a418c422f82b128f P 02d556b947994df88dda40ff5e74c9bc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 04:02:15.949402 19463 raft_consensus.cc:348] T 84247e15954e4152a418c422f82b128f P 02d556b947994df88dda40ff5e74c9bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "807d91621ef74150ba3d157c8596d3f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44747 } } peers { permanent_uuid: "9fb0406cbed74da2b933887dd8cbcac0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34235 } } peers { permanent_uuid: "02d556b947994df88dda40ff5e74c9bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43179 } }
I20260926 04:02:15.949563 19463 raft_consensus.cc:374] T 84247e15954e4152a418c422f82b128f P 02d556b947994df88dda40ff5e74c9bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:02:15.949608 19463 raft_consensus.cc:729] T 84247e15954e4152a418c422f82b128f P 02d556b947994df88dda40ff5e74c9bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02d556b947994df88dda40ff5e74c9bc, State: Initialized, Role: FOLLOWER
I20260926 04:02:15.949762 19463 consensus_queue.cc:260] T 84247e15954e4152a418c422f82b128f P 02d556b947994df88dda40ff5e74c9bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "807d91621ef74150ba3d157c8596d3f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44747 } } peers { permanent_uuid: "9fb0406cbed74da2b933887dd8cbcac0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34235 } } peers { permanent_uuid: "02d556b947994df88dda40ff5e74c9bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43179 } }
I20260926 04:02:15.949786 19462 raft_consensus.cc:348] T 84247e15954e4152a418c422f82b128f P 807d91621ef74150ba3d157c8596d3f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "807d91621ef74150ba3d157c8596d3f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44747 } } peers { permanent_uuid: "9fb0406cbed74da2b933887dd8cbcac0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34235 } } peers { permanent_uuid: "02d556b947994df88dda40ff5e74c9bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43179 } }
I20260926 04:02:15.949774 19461 raft_consensus.cc:348] T 84247e15954e4152a418c422f82b128f P 9fb0406cbed74da2b933887dd8cbcac0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "807d91621ef74150ba3d157c8596d3f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44747 } } peers { permanent_uuid: "9fb0406cbed74da2b933887dd8cbcac0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34235 } } peers { permanent_uuid: "02d556b947994df88dda40ff5e74c9bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43179 } }
I20260926 04:02:15.949946 19462 raft_consensus.cc:374] T 84247e15954e4152a418c422f82b128f P 807d91621ef74150ba3d157c8596d3f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:02:15.949946 19461 raft_consensus.cc:374] T 84247e15954e4152a418c422f82b128f P 9fb0406cbed74da2b933887dd8cbcac0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:02:15.949993 19462 raft_consensus.cc:729] T 84247e15954e4152a418c422f82b128f P 807d91621ef74150ba3d157c8596d3f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 807d91621ef74150ba3d157c8596d3f5, State: Initialized, Role: FOLLOWER
I20260926 04:02:15.949993 19461 raft_consensus.cc:729] T 84247e15954e4152a418c422f82b128f P 9fb0406cbed74da2b933887dd8cbcac0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fb0406cbed74da2b933887dd8cbcac0, State: Initialized, Role: FOLLOWER
I20260926 04:02:15.950033 19463 ts_tablet_manager.cc:1428] T 84247e15954e4152a418c422f82b128f P 02d556b947994df88dda40ff5e74c9bc: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 04:02:15.950109 19461 consensus_queue.cc:260] T 84247e15954e4152a418c422f82b128f P 9fb0406cbed74da2b933887dd8cbcac0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "807d91621ef74150ba3d157c8596d3f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44747 } } peers { permanent_uuid: "9fb0406cbed74da2b933887dd8cbcac0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34235 } } peers { permanent_uuid: "02d556b947994df88dda40ff5e74c9bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43179 } }
I20260926 04:02:15.950109 19462 consensus_queue.cc:260] T 84247e15954e4152a418c422f82b128f P 807d91621ef74150ba3d157c8596d3f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "807d91621ef74150ba3d157c8596d3f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44747 } } peers { permanent_uuid: "9fb0406cbed74da2b933887dd8cbcac0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34235 } } peers { permanent_uuid: "02d556b947994df88dda40ff5e74c9bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43179 } }
I20260926 04:02:15.950167 19185 heartbeater.cc:503] Master 127.0.98.62:39963 was elected leader, sending a full tablet report...
I20260926 04:02:15.950366 19462 ts_tablet_manager.cc:1428] T 84247e15954e4152a418c422f82b128f P 807d91621ef74150ba3d157c8596d3f5: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260926 04:02:15.950366 19461 ts_tablet_manager.cc:1428] T 84247e15954e4152a418c422f82b128f P 9fb0406cbed74da2b933887dd8cbcac0: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260926 04:02:15.950423 19447 heartbeater.cc:503] Master 127.0.98.62:39963 was elected leader, sending a full tablet report...
I20260926 04:02:15.950477 19316 heartbeater.cc:503] Master 127.0.98.62:39963 was elected leader, sending a full tablet report...
W20260926 04:02:16.017443 19317 tablet.cc:2369] T 84247e15954e4152a418c422f82b128f P 807d91621ef74150ba3d157c8596d3f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 04:02:16.058908 19469 raft_consensus.cc:482] T 84247e15954e4152a418c422f82b128f P 807d91621ef74150ba3d157c8596d3f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 04:02:16.059027 19469 raft_consensus.cc:504] T 84247e15954e4152a418c422f82b128f P 807d91621ef74150ba3d157c8596d3f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "807d91621ef74150ba3d157c8596d3f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44747 } } peers { permanent_uuid: "9fb0406cbed74da2b933887dd8cbcac0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34235 } } peers { permanent_uuid: "02d556b947994df88dda40ff5e74c9bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43179 } }
I20260926 04:02:16.059441 19469 leader_election.cc:288] T 84247e15954e4152a418c422f82b128f P 807d91621ef74150ba3d157c8596d3f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9fb0406cbed74da2b933887dd8cbcac0 (127.0.98.3:34235), 02d556b947994df88dda40ff5e74c9bc (127.0.98.1:43179)
I20260926 04:02:16.062660 19458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:16.059740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32979 (local address 127.0.98.3:34235)
0926 04:02:16.059799 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:16.059802 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:16.059812 (+ 10us) server_negotiation.cc:407] Connection header received
0926 04:02:16.059846 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:16.059847 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:16.059858 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:16.059944 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:02:16.060184 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:16.060803 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:16.061865 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:16.062260 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:16.062361 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:16.062372 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:16.062536 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 04:02:16.062576 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 04:02:16.062664 19318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:16.059657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34235 (local address 127.0.98.2:32979)
0926 04:02:16.059722 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:16.059735 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 04:02:16.059781 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:02:16.060005 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:02:16.060006 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:02:16.060008 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:02:16.060089 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:16.060091 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:16.060943 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:16.060944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:16.061743 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:16.061746 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:16.062364 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:16.062366 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:16.062443 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:16.062446 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 04:02:16.062528 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 04:02:16.062574 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 04:02:16.062999 19470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:16.059785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43179 (local address 127.0.98.2:47075)
0926 04:02:16.059974 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:16.059986 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 04:02:16.060034 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:02:16.060705 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:02:16.060706 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:02:16.060708 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:02:16.060771 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:16.060772 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:16.061421 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:16.061422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:16.062174 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:16.062176 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:16.062727 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:16.062728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:16.062809 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:16.062813 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 04:02:16.062870 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 04:02:16.062910 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":102,"threads_started":1}
I20260926 04:02:16.063006 19460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:16.059960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47075 (local address 127.0.98.1:43179)
0926 04:02:16.060045 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:16.060047 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:16.060059 (+ 12us) server_negotiation.cc:407] Connection header received
0926 04:02:16.060093 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:16.060094 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:16.060104 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:16.060171 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:02:16.060865 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:16.061334 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:16.062278 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:16.062643 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:16.062724 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:16.062732 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:16.062893 (+ 161us) server_negotiation.cc:299] Negotiation successful
0926 04:02:16.062939 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 04:02:16.062969 19401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84247e15954e4152a418c422f82b128f" candidate_uuid: "807d91621ef74150ba3d157c8596d3f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fb0406cbed74da2b933887dd8cbcac0" is_pre_election: true
I20260926 04:02:16.063127 19401 raft_consensus.cc:2454] T 84247e15954e4152a418c422f82b128f P 9fb0406cbed74da2b933887dd8cbcac0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 807d91621ef74150ba3d157c8596d3f5 in term 0.
I20260926 04:02:16.063367 19203 leader_election.cc:302] T 84247e15954e4152a418c422f82b128f P 807d91621ef74150ba3d157c8596d3f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 807d91621ef74150ba3d157c8596d3f5, 9fb0406cbed74da2b933887dd8cbcac0; no voters:
I20260926 04:02:16.063390 19139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84247e15954e4152a418c422f82b128f" candidate_uuid: "807d91621ef74150ba3d157c8596d3f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02d556b947994df88dda40ff5e74c9bc" is_pre_election: true
I20260926 04:02:16.063525 19139 raft_consensus.cc:2454] T 84247e15954e4152a418c422f82b128f P 02d556b947994df88dda40ff5e74c9bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 807d91621ef74150ba3d157c8596d3f5 in term 0.
I20260926 04:02:16.063526 19469 raft_consensus.cc:2781] T 84247e15954e4152a418c422f82b128f P 807d91621ef74150ba3d157c8596d3f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 04:02:16.063596 19469 raft_consensus.cc:482] T 84247e15954e4152a418c422f82b128f P 807d91621ef74150ba3d157c8596d3f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 04:02:16.063635 19469 raft_consensus.cc:3037] T 84247e15954e4152a418c422f82b128f P 807d91621ef74150ba3d157c8596d3f5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:02:16.064394 19469 raft_consensus.cc:504] T 84247e15954e4152a418c422f82b128f P 807d91621ef74150ba3d157c8596d3f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "807d91621ef74150ba3d157c8596d3f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44747 } } peers { permanent_uuid: "9fb0406cbed74da2b933887dd8cbcac0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34235 } } peers { permanent_uuid: "02d556b947994df88dda40ff5e74c9bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43179 } }
I20260926 04:02:16.064556 19469 leader_election.cc:288] T 84247e15954e4152a418c422f82b128f P 807d91621ef74150ba3d157c8596d3f5 [CANDIDATE]: Term 1 election: Requested vote from peers 9fb0406cbed74da2b933887dd8cbcac0 (127.0.98.3:34235), 02d556b947994df88dda40ff5e74c9bc (127.0.98.1:43179)
I20260926 04:02:16.064793 19401 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84247e15954e4152a418c422f82b128f" candidate_uuid: "807d91621ef74150ba3d157c8596d3f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9fb0406cbed74da2b933887dd8cbcac0"
I20260926 04:02:16.064841 19139 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84247e15954e4152a418c422f82b128f" candidate_uuid: "807d91621ef74150ba3d157c8596d3f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02d556b947994df88dda40ff5e74c9bc"
I20260926 04:02:16.064910 19401 raft_consensus.cc:3037] T 84247e15954e4152a418c422f82b128f P 9fb0406cbed74da2b933887dd8cbcac0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:02:16.064904 19139 raft_consensus.cc:3037] T 84247e15954e4152a418c422f82b128f P 02d556b947994df88dda40ff5e74c9bc [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:02:16.065672 19401 raft_consensus.cc:2454] T 84247e15954e4152a418c422f82b128f P 9fb0406cbed74da2b933887dd8cbcac0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 807d91621ef74150ba3d157c8596d3f5 in term 1.
I20260926 04:02:16.065685 19139 raft_consensus.cc:2454] T 84247e15954e4152a418c422f82b128f P 02d556b947994df88dda40ff5e74c9bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 807d91621ef74150ba3d157c8596d3f5 in term 1.
I20260926 04:02:16.065848 19203 leader_election.cc:302] T 84247e15954e4152a418c422f82b128f P 807d91621ef74150ba3d157c8596d3f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02d556b947994df88dda40ff5e74c9bc, 807d91621ef74150ba3d157c8596d3f5; no voters:
I20260926 04:02:16.065985 19469 raft_consensus.cc:2781] T 84247e15954e4152a418c422f82b128f P 807d91621ef74150ba3d157c8596d3f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:02:16.066164 19469 raft_consensus.cc:686] T 84247e15954e4152a418c422f82b128f P 807d91621ef74150ba3d157c8596d3f5 [term 1 LEADER]: Becoming Leader. State: Replica: 807d91621ef74150ba3d157c8596d3f5, State: Running, Role: LEADER
I20260926 04:02:16.066270 19469 consensus_queue.cc:237] T 84247e15954e4152a418c422f82b128f P 807d91621ef74150ba3d157c8596d3f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "807d91621ef74150ba3d157c8596d3f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44747 } } peers { permanent_uuid: "9fb0406cbed74da2b933887dd8cbcac0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34235 } } peers { permanent_uuid: "02d556b947994df88dda40ff5e74c9bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43179 } }
I20260926 04:02:16.067101 18997 catalog_manager.cc:5495] T 84247e15954e4152a418c422f82b128f P 807d91621ef74150ba3d157c8596d3f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 807d91621ef74150ba3d157c8596d3f5 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "807d91621ef74150ba3d157c8596d3f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "807d91621ef74150ba3d157c8596d3f5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9fb0406cbed74da2b933887dd8cbcac0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02d556b947994df88dda40ff5e74c9bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43179 } health_report { overall_health: UNKNOWN } } }
I20260926 04:02:16.089140 19457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:16.086935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52322 (local address 127.0.98.2:44747)
0926 04:02:16.086989 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:16.086990 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:16.086998 (+ 8us) server_negotiation.cc:407] Connection header received
0926 04:02:16.087051 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:16.087051 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:16.087071 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:16.087136 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 04:02:16.087260 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:16.087719 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:16.088438 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:16.088660 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:16.088711 (+ 51us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:16.088860 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 04:02:16.088943 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 04:02:16.089019 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:16.089062 (+ 43us) server_negotiation.cc:299] Negotiation successful
0926 04:02:16.089072 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 04:02:16.091194 19401 raft_consensus.cc:1261] T 84247e15954e4152a418c422f82b128f P 9fb0406cbed74da2b933887dd8cbcac0 [term 1 FOLLOWER]: Refusing update from remote peer 807d91621ef74150ba3d157c8596d3f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:02:16.091200 19139 raft_consensus.cc:1261] T 84247e15954e4152a418c422f82b128f P 02d556b947994df88dda40ff5e74c9bc [term 1 FOLLOWER]: Refusing update from remote peer 807d91621ef74150ba3d157c8596d3f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:02:16.091585 19471 consensus_queue.cc:1035] T 84247e15954e4152a418c422f82b128f P 807d91621ef74150ba3d157c8596d3f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "02d556b947994df88dda40ff5e74c9bc" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:02:16.091722 19469 consensus_queue.cc:1035] T 84247e15954e4152a418c422f82b128f P 807d91621ef74150ba3d157c8596d3f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fb0406cbed74da2b933887dd8cbcac0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:02:16.094333 19476 mvcc.cc:205] Tried to move back new op lower bound from 7333459296626241536 to 7333459296527982592. Current Snapshot: MvccSnapshot[applied={T|T < 7333459296626241536}]
I20260926 04:02:16.094430 19475 mvcc.cc:205] Tried to move back new op lower bound from 7333459296626241536 to 7333459296527982592. Current Snapshot: MvccSnapshot[applied={T|T < 7333459296626241536}]
I20260926 04:02:16.096755 18996 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41810:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 04:02:16.096869 18996 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41810:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 04:02:16.097659 18996 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4d1787dcec644fbf91d03f66edd3e3d8: Sending DeleteTablet for 3 replicas of tablet 84247e15954e4152a418c422f82b128f
I20260926 04:02:16.098076 19119 tablet_service.cc:1514] Processing DeleteTablet for tablet 84247e15954e4152a418c422f82b128f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:02:16 UTC) from {username='slave'} at 127.0.0.1:51372
I20260926 04:02:16.098076 19381 tablet_service.cc:1514] Processing DeleteTablet for tablet 84247e15954e4152a418c422f82b128f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:02:16 UTC) from {username='slave'} at 127.0.0.1:60424
I20260926 04:02:16.098310 19250 tablet_service.cc:1514] Processing DeleteTablet for tablet 84247e15954e4152a418c422f82b128f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:02:16 UTC) from {username='slave'} at 127.0.0.1:52312
I20260926 04:02:16.098500 19486 tablet_replica.cc:330] T 84247e15954e4152a418c422f82b128f P 807d91621ef74150ba3d157c8596d3f5: stopping tablet replica
I20260926 04:02:16.098516 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 19057
I20260926 04:02:16.098589 19486 raft_consensus.cc:2229] T 84247e15954e4152a418c422f82b128f P 807d91621ef74150ba3d157c8596d3f5 [term 1 LEADER]: Raft consensus shutting down.
I20260926 04:02:16.098707 19486 raft_consensus.cc:2258] T 84247e15954e4152a418c422f82b128f P 807d91621ef74150ba3d157c8596d3f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:02:16.098857 19485 tablet_replica.cc:330] T 84247e15954e4152a418c422f82b128f P 9fb0406cbed74da2b933887dd8cbcac0: stopping tablet replica
I20260926 04:02:16.099162 19485 raft_consensus.cc:2229] T 84247e15954e4152a418c422f82b128f P 9fb0406cbed74da2b933887dd8cbcac0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 04:02:16.099285 19485 raft_consensus.cc:2258] T 84247e15954e4152a418c422f82b128f P 9fb0406cbed74da2b933887dd8cbcac0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:02:16.099722 19486 ts_tablet_manager.cc:1905] T 84247e15954e4152a418c422f82b128f P 807d91621ef74150ba3d157c8596d3f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:02:16.101331 19486 ts_tablet_manager.cc:1918] T 84247e15954e4152a418c422f82b128f P 807d91621ef74150ba3d157c8596d3f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 04:02:16.101403 19486 log.cc:1196] T 84247e15954e4152a418c422f82b128f P 807d91621ef74150ba3d157c8596d3f5: Deleting WAL directory at /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395095474643-392-0/minicluster-data/ts-1/wal/wals/84247e15954e4152a418c422f82b128f
I20260926 04:02:16.101657 19486 ts_tablet_manager.cc:1939] T 84247e15954e4152a418c422f82b128f P 807d91621ef74150ba3d157c8596d3f5: Deleting consensus metadata
I20260926 04:02:16.102103 18984 catalog_manager.cc:4841] TS 807d91621ef74150ba3d157c8596d3f5 (127.0.98.2:44747): tablet 84247e15954e4152a418c422f82b128f (table test-table [id=2a469236fa05442e9f862064ce441013]) successfully deleted
I20260926 04:02:16.102367 19485 ts_tablet_manager.cc:1905] T 84247e15954e4152a418c422f82b128f P 9fb0406cbed74da2b933887dd8cbcac0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:02:16.103928 19485 ts_tablet_manager.cc:1918] T 84247e15954e4152a418c422f82b128f P 9fb0406cbed74da2b933887dd8cbcac0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 04:02:16.104003 19485 log.cc:1196] T 84247e15954e4152a418c422f82b128f P 9fb0406cbed74da2b933887dd8cbcac0: Deleting WAL directory at /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790395095474643-392-0/minicluster-data/ts-2/wal/wals/84247e15954e4152a418c422f82b128f
I20260926 04:02:16.104259 19485 ts_tablet_manager.cc:1939] T 84247e15954e4152a418c422f82b128f P 9fb0406cbed74da2b933887dd8cbcac0: Deleting consensus metadata
I20260926 04:02:16.104743 18981 catalog_manager.cc:4841] TS 9fb0406cbed74da2b933887dd8cbcac0 (127.0.98.3:34235): tablet 84247e15954e4152a418c422f82b128f (table test-table [id=2a469236fa05442e9f862064ce441013]) successfully deleted
W20260926 04:02:16.105401 18981 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43179 (error 108)
I20260926 04:02:16.105935 19456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:16.105795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43179 (local address 127.0.0.1:51384)
0926 04:02:16.105858 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:16.105900 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 04:02:16.105997 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 19188
W20260926 04:02:16.109288 18981 catalog_manager.cc:4584] TS 02d556b947994df88dda40ff5e74c9bc (127.0.98.1:43179): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 84247e15954e4152a418c422f82b128f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43179: connect: Connection refused (error 111)
I20260926 04:02:16.111441 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 19319
I20260926 04:02:16.116294 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 18966
2026-09-26T04:02:16Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (786 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 04:02:16 dist-test-slave-nhpw krb5kdc[19490](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 04:02:16 dist-test-slave-nhpw krb5kdc[19490](info): set up 2 sockets
Sep 26 04:02:16 dist-test-slave-nhpw krb5kdc[19490](info): commencing operation
krb5kdc: starting...
W20260926 04:02:17.164885 392 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/krb5kdc/test-user.keytab.
Sep 26 04:02:17 dist-test-slave-nhpw krb5kdc[19490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T04:02:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:02:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 04:02:17.283879 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41513
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41513
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:02:17.364547 19503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:02:17.364768 19503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:02:17.364791 19503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:02:17.365862 19503 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:02:17.365896 19503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:02:17.365911 19503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:02:17.365923 19503 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:02:17.365942 19503 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:02:17.367308 19503 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41513
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:02:17.367486 19503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:02:17.367647 19503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:02:17.372460 19511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:02:17.372493 19509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:02:17.372519 19508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:02:17.372737 19503 server_base.cc:1034] running on GCE node
I20260926 04:02:17.372980 19503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:02:17.373268 19503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:02:17.374459 19503 hybrid_clock.cc:648] HybridClock initialized: now 1790395337374445 us; error 29 us; skew 500 ppm
Sep 26 04:02:17 dist-test-slave-nhpw krb5kdc[19490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395337, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:02:17.377238 19503 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 04:02:17.377722 19503 webserver.cc:460] Webserver started at http://127.0.98.62:39167/ using document root <none> and password file <none>
I20260926 04:02:17.377957 19503 fs_manager.cc:359] Metadata directory not provided
I20260926 04:02:17.378026 19503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:02:17.378188 19503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:02:17.379249 19503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/minicluster-data/master-0/data/instance:
uuid: "c79cdce2ccba44cd9fd00844f197eac8"
format_stamp: "Formatted at 2026-09-26 04:02:17 on dist-test-slave-nhpw"
server_key: "4b06aaa4c44027a7e44ff1e0ead5b180"
server_key_iv: "cef26f067188e56e81ca614c85b246d9"
server_key_version: "encryptionkey@0"
I20260926 04:02:17.379618 19503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/minicluster-data/master-0/wal/instance:
uuid: "c79cdce2ccba44cd9fd00844f197eac8"
format_stamp: "Formatted at 2026-09-26 04:02:17 on dist-test-slave-nhpw"
server_key: "4b06aaa4c44027a7e44ff1e0ead5b180"
server_key_iv: "cef26f067188e56e81ca614c85b246d9"
server_key_version: "encryptionkey@0"
I20260926 04:02:17.380973 19503 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 04:02:17.381692 19518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:17.381860 19503 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 04:02:17.381968 19503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/minicluster-data/master-0/wal
uuid: "c79cdce2ccba44cd9fd00844f197eac8"
format_stamp: "Formatted at 2026-09-26 04:02:17 on dist-test-slave-nhpw"
server_key: "4b06aaa4c44027a7e44ff1e0ead5b180"
server_key_iv: "cef26f067188e56e81ca614c85b246d9"
server_key_version: "encryptionkey@0"
I20260926 04:02:17.382040 19503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:02:17.398540 19503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:02:17.400753 19503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:02:17.400915 19503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:02:17.404884 19503 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41513
I20260926 04:02:17.404915 19570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41513 every 8 connection(s)
I20260926 04:02:17.405267 19503 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
I20260926 04:02:17.406095 19571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:02:17.408231 19571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c79cdce2ccba44cd9fd00844f197eac8: Bootstrap starting.
I20260926 04:02:17.408752 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 19503
I20260926 04:02:17.408864 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/minicluster-data/master-0/wal/instance
I20260926 04:02:17.408896 19571 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c79cdce2ccba44cd9fd00844f197eac8: Neither blocks nor log segments found. Creating new log.
I20260926 04:02:17.409083 392 external_mini_cluster.cc:1428] Setting key 612c808eee6a0d8dce65dbcac0ff9baa
I20260926 04:02:17.409220 19571 log.cc:824] T 00000000000000000000000000000000 P c79cdce2ccba44cd9fd00844f197eac8: Log is configured to *not* fsync() on all Append() calls
I20260926 04:02:17.409996 19571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c79cdce2ccba44cd9fd00844f197eac8: No bootstrap required, opened a new log
I20260926 04:02:17.412236 19571 raft_consensus.cc:348] T 00000000000000000000000000000000 P c79cdce2ccba44cd9fd00844f197eac8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c79cdce2ccba44cd9fd00844f197eac8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41513 } }
I20260926 04:02:17.412403 19571 raft_consensus.cc:374] T 00000000000000000000000000000000 P c79cdce2ccba44cd9fd00844f197eac8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:02:17.412439 19571 raft_consensus.cc:729] T 00000000000000000000000000000000 P c79cdce2ccba44cd9fd00844f197eac8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c79cdce2ccba44cd9fd00844f197eac8, State: Initialized, Role: FOLLOWER
I20260926 04:02:17.412554 19571 consensus_queue.cc:260] T 00000000000000000000000000000000 P c79cdce2ccba44cd9fd00844f197eac8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c79cdce2ccba44cd9fd00844f197eac8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41513 } }
I20260926 04:02:17.412657 19571 raft_consensus.cc:388] T 00000000000000000000000000000000 P c79cdce2ccba44cd9fd00844f197eac8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:02:17.412688 19571 raft_consensus.cc:482] T 00000000000000000000000000000000 P c79cdce2ccba44cd9fd00844f197eac8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:02:17.412734 19571 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c79cdce2ccba44cd9fd00844f197eac8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:02:17.413376 19571 raft_consensus.cc:504] T 00000000000000000000000000000000 P c79cdce2ccba44cd9fd00844f197eac8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c79cdce2ccba44cd9fd00844f197eac8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41513 } }
I20260926 04:02:17.413461 19571 leader_election.cc:302] T 00000000000000000000000000000000 P c79cdce2ccba44cd9fd00844f197eac8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c79cdce2ccba44cd9fd00844f197eac8; no voters:
I20260926 04:02:17.413677 19571 leader_election.cc:288] T 00000000000000000000000000000000 P c79cdce2ccba44cd9fd00844f197eac8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 04:02:17.413705 19576 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c79cdce2ccba44cd9fd00844f197eac8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:02:17.413812 19576 raft_consensus.cc:686] T 00000000000000000000000000000000 P c79cdce2ccba44cd9fd00844f197eac8 [term 1 LEADER]: Becoming Leader. State: Replica: c79cdce2ccba44cd9fd00844f197eac8, State: Running, Role: LEADER
I20260926 04:02:17.413892 19576 consensus_queue.cc:237] T 00000000000000000000000000000000 P c79cdce2ccba44cd9fd00844f197eac8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c79cdce2ccba44cd9fd00844f197eac8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41513 } }
I20260926 04:02:17.413932 19571 sys_catalog.cc:564] T 00000000000000000000000000000000 P c79cdce2ccba44cd9fd00844f197eac8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:02:17.414196 19576 sys_catalog.cc:455] T 00000000000000000000000000000000 P c79cdce2ccba44cd9fd00844f197eac8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c79cdce2ccba44cd9fd00844f197eac8. Latest consensus state: current_term: 1 leader_uuid: "c79cdce2ccba44cd9fd00844f197eac8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c79cdce2ccba44cd9fd00844f197eac8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41513 } } }
I20260926 04:02:17.414316 19576 sys_catalog.cc:458] T 00000000000000000000000000000000 P c79cdce2ccba44cd9fd00844f197eac8 [sys.catalog]: This master's current role is: LEADER
I20260926 04:02:17.414211 19577 sys_catalog.cc:455] T 00000000000000000000000000000000 P c79cdce2ccba44cd9fd00844f197eac8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c79cdce2ccba44cd9fd00844f197eac8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c79cdce2ccba44cd9fd00844f197eac8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41513 } } }
I20260926 04:02:17.414531 19577 sys_catalog.cc:458] T 00000000000000000000000000000000 P c79cdce2ccba44cd9fd00844f197eac8 [sys.catalog]: This master's current role is: LEADER
I20260926 04:02:17.414626 19582 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:02:17.415143 19582 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:02:17.416785 19582 catalog_manager.cc:1347] Generated new cluster ID: 865ed3dc2cc64a009ac88d3a6a2725c7
I20260926 04:02:17.416836 19582 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 04:02:17 dist-test-slave-nhpw krb5kdc[19490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 04:02:17.421237 19575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:17.409570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39776 (local address 127.0.98.62:41513)
0926 04:02:17.409737 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:17.409746 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:17.409782 (+ 36us) server_negotiation.cc:407] Connection header received
0926 04:02:17.410291 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:17.410298 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:17.410344 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:17.410461 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:17.411244 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:17.412144 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:17.413221 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:17.413745 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:17.414157 (+ 412us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:17.417425 (+ 3268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:17.417429 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:17.417430 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:02:17.417441 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:17.419313 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:17.419670 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:17.419672 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:17.419673 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:17.419760 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:17.419928 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:17.419929 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:17.419929 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:17.420201 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:17.420326 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:17.420757 (+ 431us) server_negotiation.cc:299] Negotiation successful
0926 04:02:17.420882 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":72,"threads_started":1}
I20260926 04:02:17.428996 19582 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:02:17.429509 19582 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:02:17.440250 19582 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c79cdce2ccba44cd9fd00844f197eac8: Generated new TSK 0
I20260926 04:02:17.440451 19582 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260926 04:02:17.467732 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41513
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:02:17.549666 19597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:02:17.549873 19597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:02:17.549897 19597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:02:17.551091 19597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:02:17.551137 19597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:02:17.551164 19597 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 04:02:17.552805 19597 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19597
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:02:17.553061 19597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:02:17.553318 19597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:02:17.558439 19602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:02:17.558429 19603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:02:17.558426 19605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:02:17.558674 19597 server_base.cc:1034] running on GCE node
I20260926 04:02:17.558915 19597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:02:17.559127 19597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:02:17.560281 19597 hybrid_clock.cc:648] HybridClock initialized: now 1790395337560267 us; error 31 us; skew 500 ppm
Sep 26 04:02:17 dist-test-slave-nhpw krb5kdc[19490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395337, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:02:17.562920 19597 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260926 04:02:17.563431 19597 webserver.cc:460] Webserver started at http://127.0.98.1:44033/ using document root <none> and password file <none>
I20260926 04:02:17.563669 19597 fs_manager.cc:359] Metadata directory not provided
I20260926 04:02:17.563714 19597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:02:17.563879 19597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:02:17.565039 19597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/minicluster-data/ts-0/data/instance:
uuid: "5140e4bba0e5414b9907099e640cf329"
format_stamp: "Formatted at 2026-09-26 04:02:17 on dist-test-slave-nhpw"
server_key: "96f2ac29e4c9f59c500196e58e32b1f2"
server_key_iv: "e6ce99c7d2b090adaf9feb1aa39158e7"
server_key_version: "encryptionkey@0"
I20260926 04:02:17.565388 19597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5140e4bba0e5414b9907099e640cf329"
format_stamp: "Formatted at 2026-09-26 04:02:17 on dist-test-slave-nhpw"
server_key: "96f2ac29e4c9f59c500196e58e32b1f2"
server_key_iv: "e6ce99c7d2b090adaf9feb1aa39158e7"
server_key_version: "encryptionkey@0"
I20260926 04:02:17.566809 19597 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:02:17.567696 19612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:17.567914 19597 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:02:17.568030 19597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/minicluster-data/ts-0/wal
uuid: "5140e4bba0e5414b9907099e640cf329"
format_stamp: "Formatted at 2026-09-26 04:02:17 on dist-test-slave-nhpw"
server_key: "96f2ac29e4c9f59c500196e58e32b1f2"
server_key_iv: "e6ce99c7d2b090adaf9feb1aa39158e7"
server_key_version: "encryptionkey@0"
I20260926 04:02:17.568113 19597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:02:17.591334 19597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:02:17.593653 19597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:02:17.593842 19597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:02:17.594099 19597 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:02:17.594440 19597 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:02:17.594515 19597 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:17.594558 19597 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:02:17.594605 19597 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:17.600421 19597 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45723
I20260926 04:02:17.600466 19725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45723 every 8 connection(s)
I20260926 04:02:17.600839 19597 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
I20260926 04:02:17.602723 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 19597
I20260926 04:02:17.602923 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/minicluster-data/ts-0/wal/instance
I20260926 04:02:17.603147 392 external_mini_cluster.cc:1428] Setting key bcd88603cee3dfb67a2bbccfa4189bd8
Sep 26 04:02:17 dist-test-slave-nhpw krb5kdc[19490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395337, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 04:02:17.611523 19728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:17.601591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41513 (local address 127.0.98.1:52987)
0926 04:02:17.601760 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:17.601804 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 04:02:17.602715 (+ 911us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:02:17.603058 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:02:17.603062 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:02:17.603441 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:02:17.603658 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:17.603663 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:17.604436 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:17.604437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:17.605339 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:17.605342 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:17.605744 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:17.605745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:17.605877 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:17.606470 (+ 593us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:02:17.606481 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:02:17.608174 (+ 1693us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:02:17.610093 (+ 1919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:02:17.610100 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:02:17.610101 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:02:17.610327 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:02:17.610575 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:02:17.610576 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:02:17.610577 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:02:17.610689 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:02:17.610991 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:02:17.610996 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:02:17.611134 (+ 138us) client_negotiation.cc:769] Sending connection context
0926 04:02:17.611227 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 04:02:17.611354 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":103,"threads_started":1}
I20260926 04:02:17.611511 19575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:17.601615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52987 (local address 127.0.98.62:41513)
0926 04:02:17.601701 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:17.601703 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:17.601932 (+ 229us) server_negotiation.cc:407] Connection header received
0926 04:02:17.602850 (+ 918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:17.602852 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:17.602868 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:17.602956 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:17.603787 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:17.604332 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:17.605443 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:17.605660 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:17.605738 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:17.608273 (+ 2535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:17.608274 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:17.608275 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:02:17.608283 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:17.610001 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:17.610408 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:17.610410 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:17.610410 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:17.610473 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:17.610771 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:17.610772 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:17.610772 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:17.610915 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:17.610997 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:17.611277 (+ 280us) server_negotiation.cc:299] Negotiation successful
0926 04:02:17.611378 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260926 04:02:17.612143 19726 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41513
I20260926 04:02:17.612263 19726 heartbeater.cc:464] Registering TS with master...
I20260926 04:02:17.612464 19726 heartbeater.cc:511] Master 127.0.98.62:41513 requested a full tablet report, sending...
I20260926 04:02:17.613036 19535 ts_manager.cc:194] Registered new tserver with Master: 5140e4bba0e5414b9907099e640cf329 (127.0.98.1:45723)
I20260926 04:02:17.613662 19535 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52987
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260926 04:02:17.631620 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41513
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:02:17.711787 19731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:02:17.711993 19731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:02:17.712016 19731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:02:17.713110 19731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:02:17.713145 19731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:02:17.713171 19731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 04:02:17.714787 19731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:02:17.714965 19731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:02:17.715193 19731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:02:17.720002 19736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:02:17.720029 19737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:02:17.720212 19731 server_base.cc:1034] running on GCE node
W20260926 04:02:17.720120 19739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:02:17.720687 19731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:02:17.720965 19731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:02:17.722124 19731 hybrid_clock.cc:648] HybridClock initialized: now 1790395337722112 us; error 28 us; skew 500 ppm
Sep 26 04:02:17 dist-test-slave-nhpw krb5kdc[19490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395337, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:02:17.724674 19731 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260926 04:02:17.725129 19731 webserver.cc:460] Webserver started at http://127.0.98.2:44833/ using document root <none> and password file <none>
I20260926 04:02:17.725350 19731 fs_manager.cc:359] Metadata directory not provided
I20260926 04:02:17.725414 19731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:02:17.725543 19731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:02:17.726584 19731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/minicluster-data/ts-1/data/instance:
uuid: "a5e1cc8ae26c49349bde05b594170056"
format_stamp: "Formatted at 2026-09-26 04:02:17 on dist-test-slave-nhpw"
server_key: "17dc8bd48622010c23675772bc6a181e"
server_key_iv: "c8019e7adc479ab97867e1f0430dcb12"
server_key_version: "encryptionkey@0"
I20260926 04:02:17.726936 19731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a5e1cc8ae26c49349bde05b594170056"
format_stamp: "Formatted at 2026-09-26 04:02:17 on dist-test-slave-nhpw"
server_key: "17dc8bd48622010c23675772bc6a181e"
server_key_iv: "c8019e7adc479ab97867e1f0430dcb12"
server_key_version: "encryptionkey@0"
I20260926 04:02:17.728259 19731 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 04:02:17.729056 19746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:17.729225 19731 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:17.729318 19731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/minicluster-data/ts-1/wal
uuid: "a5e1cc8ae26c49349bde05b594170056"
format_stamp: "Formatted at 2026-09-26 04:02:17 on dist-test-slave-nhpw"
server_key: "17dc8bd48622010c23675772bc6a181e"
server_key_iv: "c8019e7adc479ab97867e1f0430dcb12"
server_key_version: "encryptionkey@0"
I20260926 04:02:17.729394 19731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:02:17.738384 19731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:02:17.740335 19731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:02:17.740495 19731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:02:17.740784 19731 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:02:17.741088 19731 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:02:17.741139 19731 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:17.741173 19731 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:02:17.741199 19731 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:17.747067 19731 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44217
I20260926 04:02:17.747120 19859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44217 every 8 connection(s)
I20260926 04:02:17.747421 19731 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
Sep 26 04:02:17 dist-test-slave-nhpw krb5kdc[19490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395337, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 04:02:17.756618 19575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:17.748192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43343 (local address 127.0.98.62:41513)
0926 04:02:17.748261 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:17.748263 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:17.748450 (+ 187us) server_negotiation.cc:407] Connection header received
0926 04:02:17.749446 (+ 996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:17.749447 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:17.749458 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:17.749519 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:17.750085 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:17.750581 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:17.751650 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:17.751851 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:17.751913 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:17.754096 (+ 2183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:17.754098 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:17.754098 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:02:17.754105 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:17.755429 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:17.755788 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:17.755789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:17.755789 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:17.755835 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:17.756040 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:17.756041 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:17.756041 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:17.756137 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:17.756186 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:17.756409 (+ 223us) server_negotiation.cc:299] Negotiation successful
0926 04:02:17.756507 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 04:02:17.756678 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 19731
I20260926 04:02:17.756712 19862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:17.748142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41513 (local address 127.0.98.2:43343)
0926 04:02:17.748306 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:17.748349 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 04:02:17.749333 (+ 984us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:02:17.749575 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:02:17.749578 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:02:17.749826 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:02:17.750014 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:17.750018 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:17.750684 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:17.750685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:17.751564 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:17.751566 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:17.751944 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:17.751945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:17.752055 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:17.752547 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:02:17.752555 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:02:17.754010 (+ 1455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:02:17.755504 (+ 1494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:02:17.755509 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:02:17.755511 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:02:17.755724 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:02:17.755893 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:02:17.755893 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:02:17.755894 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:02:17.755986 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:02:17.756192 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:02:17.756196 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:02:17.756303 (+ 107us) client_negotiation.cc:769] Sending connection context
0926 04:02:17.756382 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 04:02:17.756494 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":91,"threads_started":1}
I20260926 04:02:17.756779 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/minicluster-data/ts-1/wal/instance
I20260926 04:02:17.756991 392 external_mini_cluster.cc:1428] Setting key 3df6a1feac082b26094d7d5896403234
I20260926 04:02:17.757195 19860 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41513
I20260926 04:02:17.757306 19860 heartbeater.cc:464] Registering TS with master...
I20260926 04:02:17.757475 19860 heartbeater.cc:511] Master 127.0.98.62:41513 requested a full tablet report, sending...
I20260926 04:02:17.757859 19535 ts_manager.cc:194] Registered new tserver with Master: a5e1cc8ae26c49349bde05b594170056 (127.0.98.2:44217)
I20260926 04:02:17.758368 19535 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43343
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260926 04:02:17.783710 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41513
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39287
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:02:17.864792 19865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:02:17.864987 19865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:02:17.865010 19865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:02:17.866103 19865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:02:17.866137 19865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:02:17.866163 19865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 04:02:17.867533 19865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39287
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41513
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:02:17.867777 19865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:02:17.867952 19865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:02:17.872941 19870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:02:17.873050 19873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:02:17.873070 19871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:02:17.873292 19865 server_base.cc:1034] running on GCE node
I20260926 04:02:17.873544 19865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:02:17.873786 19865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:02:17.874953 19865 hybrid_clock.cc:648] HybridClock initialized: now 1790395337874942 us; error 35 us; skew 500 ppm
Sep 26 04:02:17 dist-test-slave-nhpw krb5kdc[19490](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395337, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:02:17.877556 19865 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260926 04:02:17.878072 19865 webserver.cc:460] Webserver started at http://127.0.98.3:36719/ using document root <none> and password file <none>
I20260926 04:02:17.878327 19865 fs_manager.cc:359] Metadata directory not provided
I20260926 04:02:17.878393 19865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:02:17.878542 19865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:02:17.879565 19865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/minicluster-data/ts-2/data/instance:
uuid: "e27ef819ceab487f8156060f59151530"
format_stamp: "Formatted at 2026-09-26 04:02:17 on dist-test-slave-nhpw"
server_key: "90f59a6d88cdf70c1af902209814c0d8"
server_key_iv: "a063b60c50802ce7bd390603880b5fd6"
server_key_version: "encryptionkey@0"
I20260926 04:02:17.879917 19865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e27ef819ceab487f8156060f59151530"
format_stamp: "Formatted at 2026-09-26 04:02:17 on dist-test-slave-nhpw"
server_key: "90f59a6d88cdf70c1af902209814c0d8"
server_key_iv: "a063b60c50802ce7bd390603880b5fd6"
server_key_version: "encryptionkey@0"
I20260926 04:02:17.881278 19865 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 04:02:17.882082 19880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:17.882246 19865 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 04:02:17.882344 19865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/minicluster-data/ts-2/wal
uuid: "e27ef819ceab487f8156060f59151530"
format_stamp: "Formatted at 2026-09-26 04:02:17 on dist-test-slave-nhpw"
server_key: "90f59a6d88cdf70c1af902209814c0d8"
server_key_iv: "a063b60c50802ce7bd390603880b5fd6"
server_key_version: "encryptionkey@0"
I20260926 04:02:17.882418 19865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:02:17.905807 19865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:02:17.907771 19865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:02:17.907941 19865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:02:17.908228 19865 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:02:17.908540 19865 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:02:17.908595 19865 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:17.908631 19865 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:02:17.908685 19865 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:17.914837 19865 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42555
I20260926 04:02:17.914875 19993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42555 every 8 connection(s)
I20260926 04:02:17.915256 19865 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
I20260926 04:02:17.918408 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 19865
I20260926 04:02:17.918531 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/minicluster-data/ts-2/wal/instance
I20260926 04:02:17.918720 392 external_mini_cluster.cc:1428] Setting key badfb047a2e7dd2630d3280ab23eeaf2
Sep 26 04:02:17 dist-test-slave-nhpw krb5kdc[19490](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395337, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 04:02:17.925170 19575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:17.915959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53887 (local address 127.0.98.62:41513)
0926 04:02:17.916024 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:17.916026 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:17.916243 (+ 217us) server_negotiation.cc:407] Connection header received
0926 04:02:17.917283 (+ 1040us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:17.917284 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:17.917297 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:17.917353 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:17.918028 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:17.918771 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:17.920044 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:17.920235 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:17.920294 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:17.922621 (+ 2327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:17.922623 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:17.922623 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:02:17.922630 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:17.923880 (+ 1250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:17.924298 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:17.924299 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:17.924300 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:17.924350 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:17.924560 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:17.924561 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:17.924561 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:17.924726 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:17.924780 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:17.925044 (+ 264us) server_negotiation.cc:299] Negotiation successful
0926 04:02:17.925106 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 04:02:17.925379 19996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:17.915951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41513 (local address 127.0.98.3:53887)
0926 04:02:17.916123 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:17.916158 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 04:02:17.917167 (+ 1009us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:02:17.917430 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:02:17.917434 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:02:17.917707 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:02:17.917912 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:17.917916 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:17.918883 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:17.918884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:17.919964 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:17.919967 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:17.920311 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:17.920312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:17.920410 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:17.920966 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:02:17.920977 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:02:17.922533 (+ 1556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:02:17.923986 (+ 1453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:02:17.923993 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:02:17.923995 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:02:17.924233 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:02:17.924406 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:02:17.924407 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:02:17.924407 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:02:17.924507 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:02:17.924782 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:02:17.924787 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:02:17.924903 (+ 116us) client_negotiation.cc:769] Sending connection context
0926 04:02:17.925020 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 04:02:17.925144 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":79,"threads_started":1}
I20260926 04:02:17.925899 19994 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41513
I20260926 04:02:17.926024 19994 heartbeater.cc:464] Registering TS with master...
I20260926 04:02:17.926172 19994 heartbeater.cc:511] Master 127.0.98.62:41513 requested a full tablet report, sending...
I20260926 04:02:17.926595 19535 ts_manager.cc:194] Registered new tserver with Master: e27ef819ceab487f8156060f59151530 (127.0.98.3:42555)
I20260926 04:02:17.926988 19535 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53887
I20260926 04:02:17.930229 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 04:02:17.937927 19575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:17.931622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39788 (local address 127.0.98.62:41513)
0926 04:02:17.931687 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:17.931689 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:17.931876 (+ 187us) server_negotiation.cc:407] Connection header received
0926 04:02:17.931908 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:17.931909 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:17.931922 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:17.932001 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:17.932564 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:17.933188 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:17.933904 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:17.934103 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:17.934164 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:17.935392 (+ 1228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:17.935394 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:17.935394 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:02:17.935401 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:17.936686 (+ 1285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:17.937072 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:17.937074 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:17.937074 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:17.937134 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:17.937333 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:17.937334 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:17.937335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:17.937499 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:17.937607 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:17.937720 (+ 113us) server_negotiation.cc:299] Negotiation successful
0926 04:02:17.937805 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 04:02:17.939424 19535 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39788:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 04:02:17.941390 19535 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 04:02:17.948417 20010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:17.944175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38096 (local address 127.0.98.3:42555)
0926 04:02:17.944359 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:17.944362 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:17.944370 (+ 8us) server_negotiation.cc:407] Connection header received
0926 04:02:17.944423 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:17.944425 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:17.944442 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:17.944525 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:02:17.944797 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:17.945893 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:17.947315 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:17.948047 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:17.948115 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:17.948175 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:17.948319 (+ 144us) server_negotiation.cc:299] Negotiation successful
0926 04:02:17.948352 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":84,"threads_started":1}
I20260926 04:02:17.948428 20007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:17.943888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42555 (local address 127.0.0.1:38096)
0926 04:02:17.944229 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:17.944240 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 04:02:17.944297 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:02:17.944546 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:02:17.944550 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:02:17.944552 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:02:17.944624 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:17.944626 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:17.945994 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:17.945995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:17.947222 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:17.947224 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:17.948122 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:17.948123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:17.948204 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:17.948219 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 04:02:17.948295 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 04:02:17.948346 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":89,"threads_started":1}
I20260926 04:02:17.948889 20005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:17.943889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45723 (local address 127.0.0.1:43292)
0926 04:02:17.944348 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:17.944360 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 04:02:17.944416 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:02:17.944830 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:02:17.944831 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:02:17.944832 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:02:17.944896 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:17.944898 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:17.946113 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:17.946113 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:17.947222 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:17.947224 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:17.948655 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:17.948656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:17.948724 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:17.948728 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 04:02:17.948782 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 04:02:17.948822 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":442,"mutex_wait_us":10,"thread_start_us":139,"threads_started":1}
I20260926 04:02:17.949242 20009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:17.944115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43292 (local address 127.0.98.1:45723)
0926 04:02:17.944302 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:17.944307 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:17.944438 (+ 131us) server_negotiation.cc:407] Connection header received
0926 04:02:17.944627 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:17.944653 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:17.944672 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:17.944739 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:02:17.945002 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:17.946046 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:17.947315 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:17.948037 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:17.948115 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:17.948175 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:17.948922 (+ 747us) server_negotiation.cc:299] Negotiation successful
0926 04:02:17.949157 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":79,"threads_started":1}
I20260926 04:02:17.949229 19928 tablet_service.cc:1467] Processing CreateTablet for tablet 17074549f0f94857a75c0aef38b9a99e (DEFAULT_TABLE table=test-table [id=0b61a65cb8f14327930340b13b0b350b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:02:17.949620 19928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17074549f0f94857a75c0aef38b9a99e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:02:17.949879 19660 tablet_service.cc:1467] Processing CreateTablet for tablet 17074549f0f94857a75c0aef38b9a99e (DEFAULT_TABLE table=test-table [id=0b61a65cb8f14327930340b13b0b350b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:02:17.950181 19660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17074549f0f94857a75c0aef38b9a99e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:02:17.952476 20011 tablet_bootstrap.cc:492] T 17074549f0f94857a75c0aef38b9a99e P e27ef819ceab487f8156060f59151530: Bootstrap starting.
I20260926 04:02:17.952972 20012 tablet_bootstrap.cc:492] T 17074549f0f94857a75c0aef38b9a99e P 5140e4bba0e5414b9907099e640cf329: Bootstrap starting.
I20260926 04:02:17.953042 20006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:17.943887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44217 (local address 127.0.0.1:39932)
0926 04:02:17.944083 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:17.944124 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 04:02:17.944189 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:02:17.944816 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:02:17.944817 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:02:17.944820 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:02:17.944896 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:17.944898 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:17.945804 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:17.945805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:17.947903 (+ 2098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:17.947906 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:17.952755 (+ 4849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:17.952756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:17.952845 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:17.952849 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 04:02:17.952907 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 04:02:17.952958 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":1279,"thread_start_us":91,"threads_started":1}
I20260926 04:02:17.953214 20011 tablet_bootstrap.cc:654] T 17074549f0f94857a75c0aef38b9a99e P e27ef819ceab487f8156060f59151530: Neither blocks nor log segments found. Creating new log.
I20260926 04:02:17.953301 20008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:17.944069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39932 (local address 127.0.98.2:44217)
0926 04:02:17.944531 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:17.944537 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:17.944549 (+ 12us) server_negotiation.cc:407] Connection header received
0926 04:02:17.944599 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:17.944601 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:17.944619 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:17.944721 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:02:17.944991 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:17.945698 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:17.948003 (+ 2305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:17.952549 (+ 4546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:17.952618 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:17.952721 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:17.953165 (+ 444us) server_negotiation.cc:299] Negotiation successful
0926 04:02:17.953219 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":78,"threads_started":1}
I20260926 04:02:17.953774 20012 tablet_bootstrap.cc:654] T 17074549f0f94857a75c0aef38b9a99e P 5140e4bba0e5414b9907099e640cf329: Neither blocks nor log segments found. Creating new log.
I20260926 04:02:17.953846 20011 log.cc:824] T 17074549f0f94857a75c0aef38b9a99e P e27ef819ceab487f8156060f59151530: Log is configured to *not* fsync() on all Append() calls
I20260926 04:02:17.954031 19794 tablet_service.cc:1467] Processing CreateTablet for tablet 17074549f0f94857a75c0aef38b9a99e (DEFAULT_TABLE table=test-table [id=0b61a65cb8f14327930340b13b0b350b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:02:17.954315 19794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 17074549f0f94857a75c0aef38b9a99e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:02:17.954372 20012 log.cc:824] T 17074549f0f94857a75c0aef38b9a99e P 5140e4bba0e5414b9907099e640cf329: Log is configured to *not* fsync() on all Append() calls
I20260926 04:02:17.954810 20011 tablet_bootstrap.cc:492] T 17074549f0f94857a75c0aef38b9a99e P e27ef819ceab487f8156060f59151530: No bootstrap required, opened a new log
I20260926 04:02:17.955006 20011 ts_tablet_manager.cc:1397] T 17074549f0f94857a75c0aef38b9a99e P e27ef819ceab487f8156060f59151530: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 04:02:17.955111 20012 tablet_bootstrap.cc:492] T 17074549f0f94857a75c0aef38b9a99e P 5140e4bba0e5414b9907099e640cf329: No bootstrap required, opened a new log
I20260926 04:02:17.955178 20012 ts_tablet_manager.cc:1397] T 17074549f0f94857a75c0aef38b9a99e P 5140e4bba0e5414b9907099e640cf329: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 04:02:17.956796 20012 raft_consensus.cc:348] T 17074549f0f94857a75c0aef38b9a99e P 5140e4bba0e5414b9907099e640cf329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5e1cc8ae26c49349bde05b594170056" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44217 } } peers { permanent_uuid: "5140e4bba0e5414b9907099e640cf329" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45723 } } peers { permanent_uuid: "e27ef819ceab487f8156060f59151530" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42555 } }
I20260926 04:02:17.956969 20012 raft_consensus.cc:374] T 17074549f0f94857a75c0aef38b9a99e P 5140e4bba0e5414b9907099e640cf329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:02:17.957015 20012 raft_consensus.cc:729] T 17074549f0f94857a75c0aef38b9a99e P 5140e4bba0e5414b9907099e640cf329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5140e4bba0e5414b9907099e640cf329, State: Initialized, Role: FOLLOWER
I20260926 04:02:17.957156 20012 consensus_queue.cc:260] T 17074549f0f94857a75c0aef38b9a99e P 5140e4bba0e5414b9907099e640cf329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5e1cc8ae26c49349bde05b594170056" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44217 } } peers { permanent_uuid: "5140e4bba0e5414b9907099e640cf329" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45723 } } peers { permanent_uuid: "e27ef819ceab487f8156060f59151530" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42555 } }
I20260926 04:02:17.957151 20011 raft_consensus.cc:348] T 17074549f0f94857a75c0aef38b9a99e P e27ef819ceab487f8156060f59151530 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5e1cc8ae26c49349bde05b594170056" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44217 } } peers { permanent_uuid: "5140e4bba0e5414b9907099e640cf329" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45723 } } peers { permanent_uuid: "e27ef819ceab487f8156060f59151530" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42555 } }
I20260926 04:02:17.957324 20011 raft_consensus.cc:374] T 17074549f0f94857a75c0aef38b9a99e P e27ef819ceab487f8156060f59151530 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:02:17.957365 20011 raft_consensus.cc:729] T 17074549f0f94857a75c0aef38b9a99e P e27ef819ceab487f8156060f59151530 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e27ef819ceab487f8156060f59151530, State: Initialized, Role: FOLLOWER
I20260926 04:02:17.957460 20012 ts_tablet_manager.cc:1428] T 17074549f0f94857a75c0aef38b9a99e P 5140e4bba0e5414b9907099e640cf329: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 04:02:17.957485 20011 consensus_queue.cc:260] T 17074549f0f94857a75c0aef38b9a99e P e27ef819ceab487f8156060f59151530 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5e1cc8ae26c49349bde05b594170056" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44217 } } peers { permanent_uuid: "5140e4bba0e5414b9907099e640cf329" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45723 } } peers { permanent_uuid: "e27ef819ceab487f8156060f59151530" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42555 } }
I20260926 04:02:17.957614 19726 heartbeater.cc:503] Master 127.0.98.62:41513 was elected leader, sending a full tablet report...
I20260926 04:02:17.957746 20011 ts_tablet_manager.cc:1428] T 17074549f0f94857a75c0aef38b9a99e P e27ef819ceab487f8156060f59151530: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 04:02:17.957790 19994 heartbeater.cc:503] Master 127.0.98.62:41513 was elected leader, sending a full tablet report...
I20260926 04:02:17.957741 20016 tablet_bootstrap.cc:492] T 17074549f0f94857a75c0aef38b9a99e P a5e1cc8ae26c49349bde05b594170056: Bootstrap starting.
I20260926 04:02:17.958542 20016 tablet_bootstrap.cc:654] T 17074549f0f94857a75c0aef38b9a99e P a5e1cc8ae26c49349bde05b594170056: Neither blocks nor log segments found. Creating new log.
I20260926 04:02:17.959156 20016 log.cc:824] T 17074549f0f94857a75c0aef38b9a99e P a5e1cc8ae26c49349bde05b594170056: Log is configured to *not* fsync() on all Append() calls
I20260926 04:02:17.959928 20016 tablet_bootstrap.cc:492] T 17074549f0f94857a75c0aef38b9a99e P a5e1cc8ae26c49349bde05b594170056: No bootstrap required, opened a new log
I20260926 04:02:17.960003 20016 ts_tablet_manager.cc:1397] T 17074549f0f94857a75c0aef38b9a99e P a5e1cc8ae26c49349bde05b594170056: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 04:02:17.961442 20016 raft_consensus.cc:348] T 17074549f0f94857a75c0aef38b9a99e P a5e1cc8ae26c49349bde05b594170056 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5e1cc8ae26c49349bde05b594170056" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44217 } } peers { permanent_uuid: "5140e4bba0e5414b9907099e640cf329" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45723 } } peers { permanent_uuid: "e27ef819ceab487f8156060f59151530" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42555 } }
I20260926 04:02:17.961594 20016 raft_consensus.cc:374] T 17074549f0f94857a75c0aef38b9a99e P a5e1cc8ae26c49349bde05b594170056 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:02:17.961638 20016 raft_consensus.cc:729] T 17074549f0f94857a75c0aef38b9a99e P a5e1cc8ae26c49349bde05b594170056 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5e1cc8ae26c49349bde05b594170056, State: Initialized, Role: FOLLOWER
I20260926 04:02:17.961755 20016 consensus_queue.cc:260] T 17074549f0f94857a75c0aef38b9a99e P a5e1cc8ae26c49349bde05b594170056 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5e1cc8ae26c49349bde05b594170056" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44217 } } peers { permanent_uuid: "5140e4bba0e5414b9907099e640cf329" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45723 } } peers { permanent_uuid: "e27ef819ceab487f8156060f59151530" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42555 } }
I20260926 04:02:17.962061 20016 ts_tablet_manager.cc:1428] T 17074549f0f94857a75c0aef38b9a99e P a5e1cc8ae26c49349bde05b594170056: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 04:02:17.962065 19860 heartbeater.cc:503] Master 127.0.98.62:41513 was elected leader, sending a full tablet report...
I20260926 04:02:17.992161 20019 raft_consensus.cc:482] T 17074549f0f94857a75c0aef38b9a99e P a5e1cc8ae26c49349bde05b594170056 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 04:02:17.992283 20019 raft_consensus.cc:504] T 17074549f0f94857a75c0aef38b9a99e P a5e1cc8ae26c49349bde05b594170056 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5e1cc8ae26c49349bde05b594170056" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44217 } } peers { permanent_uuid: "5140e4bba0e5414b9907099e640cf329" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45723 } } peers { permanent_uuid: "e27ef819ceab487f8156060f59151530" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42555 } }
I20260926 04:02:17.992753 20019 leader_election.cc:288] T 17074549f0f94857a75c0aef38b9a99e P a5e1cc8ae26c49349bde05b594170056 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5140e4bba0e5414b9907099e640cf329 (127.0.98.1:45723), e27ef819ceab487f8156060f59151530 (127.0.98.3:42555)
I20260926 04:02:17.996618 20009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:17.993036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37845 (local address 127.0.98.1:45723)
0926 04:02:17.993090 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:17.993092 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:17.993100 (+ 8us) server_negotiation.cc:407] Connection header received
0926 04:02:17.993131 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:17.993132 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:17.993143 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:17.993206 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:02:17.993406 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:17.994189 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:17.995671 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:17.996232 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:17.996308 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:17.996318 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:17.996508 (+ 190us) server_negotiation.cc:299] Negotiation successful
0926 04:02:17.996549 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 04:02:17.996618 20010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:17.993036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40091 (local address 127.0.98.3:42555)
0926 04:02:17.993087 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:17.993090 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:17.993242 (+ 152us) server_negotiation.cc:407] Connection header received
0926 04:02:17.993293 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:17.993294 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:17.993301 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:17.993340 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:02:17.993493 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:17.994175 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:17.995671 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:17.996232 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:17.996308 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:17.996318 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:17.996501 (+ 183us) server_negotiation.cc:299] Negotiation successful
0926 04:02:17.996552 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 04:02:17.996651 19862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:17.992952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45723 (local address 127.0.98.2:37845)
0926 04:02:17.993012 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:17.993024 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 04:02:17.993070 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:02:17.993232 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:02:17.993233 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:02:17.993235 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:02:17.993313 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:17.993315 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:17.994265 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:17.994265 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:17.995573 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:17.995576 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:17.996331 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:17.996332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:17.996419 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:17.996422 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 04:02:17.996496 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 04:02:17.996536 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":66}
I20260926 04:02:17.996657 20020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:17.992952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42555 (local address 127.0.98.2:40091)
0926 04:02:17.993176 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:17.993185 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 04:02:17.993232 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:02:17.993362 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:02:17.993363 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:02:17.993365 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:02:17.993427 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:17.993428 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:17.994263 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:17.994264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:17.995573 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:17.995576 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:17.996331 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:17.996332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:17.996419 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:17.996423 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 04:02:17.996486 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 04:02:17.996535 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":78,"thread_start_us":141,"threads_started":1}
I20260926 04:02:17.997078 19680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17074549f0f94857a75c0aef38b9a99e" candidate_uuid: "a5e1cc8ae26c49349bde05b594170056" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5140e4bba0e5414b9907099e640cf329" is_pre_election: true
I20260926 04:02:17.997273 19680 raft_consensus.cc:2454] T 17074549f0f94857a75c0aef38b9a99e P 5140e4bba0e5414b9907099e640cf329 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5e1cc8ae26c49349bde05b594170056 in term 0.
I20260926 04:02:17.997283 19948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17074549f0f94857a75c0aef38b9a99e" candidate_uuid: "a5e1cc8ae26c49349bde05b594170056" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e27ef819ceab487f8156060f59151530" is_pre_election: true
I20260926 04:02:17.997403 19948 raft_consensus.cc:2454] T 17074549f0f94857a75c0aef38b9a99e P e27ef819ceab487f8156060f59151530 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5e1cc8ae26c49349bde05b594170056 in term 0.
I20260926 04:02:17.997526 19750 leader_election.cc:302] T 17074549f0f94857a75c0aef38b9a99e P a5e1cc8ae26c49349bde05b594170056 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5140e4bba0e5414b9907099e640cf329, a5e1cc8ae26c49349bde05b594170056; no voters:
I20260926 04:02:17.997692 20019 raft_consensus.cc:2781] T 17074549f0f94857a75c0aef38b9a99e P a5e1cc8ae26c49349bde05b594170056 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 04:02:17.997731 20019 raft_consensus.cc:482] T 17074549f0f94857a75c0aef38b9a99e P a5e1cc8ae26c49349bde05b594170056 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 04:02:17.997747 20019 raft_consensus.cc:3037] T 17074549f0f94857a75c0aef38b9a99e P a5e1cc8ae26c49349bde05b594170056 [term 0 FOLLOWER]: Advancing to term 1
W20260926 04:02:17.997978 19861 tablet.cc:2369] T 17074549f0f94857a75c0aef38b9a99e P a5e1cc8ae26c49349bde05b594170056: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 04:02:17.998613 20019 raft_consensus.cc:504] T 17074549f0f94857a75c0aef38b9a99e P a5e1cc8ae26c49349bde05b594170056 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5e1cc8ae26c49349bde05b594170056" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44217 } } peers { permanent_uuid: "5140e4bba0e5414b9907099e640cf329" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45723 } } peers { permanent_uuid: "e27ef819ceab487f8156060f59151530" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42555 } }
I20260926 04:02:17.998771 20019 leader_election.cc:288] T 17074549f0f94857a75c0aef38b9a99e P a5e1cc8ae26c49349bde05b594170056 [CANDIDATE]: Term 1 election: Requested vote from peers 5140e4bba0e5414b9907099e640cf329 (127.0.98.1:45723), e27ef819ceab487f8156060f59151530 (127.0.98.3:42555)
I20260926 04:02:17.998893 19680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17074549f0f94857a75c0aef38b9a99e" candidate_uuid: "a5e1cc8ae26c49349bde05b594170056" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5140e4bba0e5414b9907099e640cf329"
I20260926 04:02:17.998951 19948 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "17074549f0f94857a75c0aef38b9a99e" candidate_uuid: "a5e1cc8ae26c49349bde05b594170056" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e27ef819ceab487f8156060f59151530"
I20260926 04:02:17.998970 19680 raft_consensus.cc:3037] T 17074549f0f94857a75c0aef38b9a99e P 5140e4bba0e5414b9907099e640cf329 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:02:17.999027 19948 raft_consensus.cc:3037] T 17074549f0f94857a75c0aef38b9a99e P e27ef819ceab487f8156060f59151530 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:02:17.999852 19948 raft_consensus.cc:2454] T 17074549f0f94857a75c0aef38b9a99e P e27ef819ceab487f8156060f59151530 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5e1cc8ae26c49349bde05b594170056 in term 1.
I20260926 04:02:17.999850 19680 raft_consensus.cc:2454] T 17074549f0f94857a75c0aef38b9a99e P 5140e4bba0e5414b9907099e640cf329 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5e1cc8ae26c49349bde05b594170056 in term 1.
I20260926 04:02:18.000057 19749 leader_election.cc:302] T 17074549f0f94857a75c0aef38b9a99e P a5e1cc8ae26c49349bde05b594170056 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a5e1cc8ae26c49349bde05b594170056, e27ef819ceab487f8156060f59151530; no voters:
I20260926 04:02:18.000195 20019 raft_consensus.cc:2781] T 17074549f0f94857a75c0aef38b9a99e P a5e1cc8ae26c49349bde05b594170056 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:02:18.000394 20019 raft_consensus.cc:686] T 17074549f0f94857a75c0aef38b9a99e P a5e1cc8ae26c49349bde05b594170056 [term 1 LEADER]: Becoming Leader. State: Replica: a5e1cc8ae26c49349bde05b594170056, State: Running, Role: LEADER
I20260926 04:02:18.000538 20019 consensus_queue.cc:237] T 17074549f0f94857a75c0aef38b9a99e P a5e1cc8ae26c49349bde05b594170056 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5e1cc8ae26c49349bde05b594170056" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44217 } } peers { permanent_uuid: "5140e4bba0e5414b9907099e640cf329" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45723 } } peers { permanent_uuid: "e27ef819ceab487f8156060f59151530" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42555 } }
I20260926 04:02:18.001467 19534 catalog_manager.cc:5495] T 17074549f0f94857a75c0aef38b9a99e P a5e1cc8ae26c49349bde05b594170056 reported cstate change: term changed from 0 to 1, leader changed from <none> to a5e1cc8ae26c49349bde05b594170056 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "a5e1cc8ae26c49349bde05b594170056" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a5e1cc8ae26c49349bde05b594170056" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44217 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5140e4bba0e5414b9907099e640cf329" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e27ef819ceab487f8156060f59151530" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42555 } health_report { overall_health: UNKNOWN } } }
I20260926 04:02:18.021064 20008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:18.018775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39940 (local address 127.0.98.2:44217)
0926 04:02:18.018859 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:18.018861 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:18.018872 (+ 11us) server_negotiation.cc:407] Connection header received
0926 04:02:18.018909 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:18.018910 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:18.018929 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:18.019000 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 04:02:18.019122 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:18.019593 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:18.020280 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:18.020463 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:18.020511 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:18.020707 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 04:02:18.020807 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 04:02:18.020871 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:18.020900 (+ 29us) server_negotiation.cc:299] Negotiation successful
0926 04:02:18.020922 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 04:02:18.022825 19680 raft_consensus.cc:1261] T 17074549f0f94857a75c0aef38b9a99e P 5140e4bba0e5414b9907099e640cf329 [term 1 FOLLOWER]: Refusing update from remote peer a5e1cc8ae26c49349bde05b594170056: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:02:18.022825 19948 raft_consensus.cc:1261] T 17074549f0f94857a75c0aef38b9a99e P e27ef819ceab487f8156060f59151530 [term 1 FOLLOWER]: Refusing update from remote peer a5e1cc8ae26c49349bde05b594170056: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:02:18.023187 20019 consensus_queue.cc:1035] T 17074549f0f94857a75c0aef38b9a99e P a5e1cc8ae26c49349bde05b594170056 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e27ef819ceab487f8156060f59151530" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:02:18.023356 20021 consensus_queue.cc:1035] T 17074549f0f94857a75c0aef38b9a99e P a5e1cc8ae26c49349bde05b594170056 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5140e4bba0e5414b9907099e640cf329" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:02:18.026044 20025 mvcc.cc:205] Tried to move back new op lower bound from 7333459304538763264 to 7333459304450928640. Current Snapshot: MvccSnapshot[applied={T|T < 7333459304538763264}]
I20260926 04:02:18.026161 20026 mvcc.cc:205] Tried to move back new op lower bound from 7333459304538763264 to 7333459304450928640. Current Snapshot: MvccSnapshot[applied={T|T < 7333459304538763264}]
I20260926 04:02:18.027942 19535 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39788:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 04:02:18.028030 19535 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39788:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 04:02:18.028756 19535 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c79cdce2ccba44cd9fd00844f197eac8: Sending DeleteTablet for 3 replicas of tablet 17074549f0f94857a75c0aef38b9a99e
I20260926 04:02:18.029085 19794 tablet_service.cc:1514] Processing DeleteTablet for tablet 17074549f0f94857a75c0aef38b9a99e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:02:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39932
I20260926 04:02:18.029096 19660 tablet_service.cc:1514] Processing DeleteTablet for tablet 17074549f0f94857a75c0aef38b9a99e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:02:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43292
I20260926 04:02:18.029316 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 19597
I20260926 04:02:18.029318 19928 tablet_service.cc:1514] Processing DeleteTablet for tablet 17074549f0f94857a75c0aef38b9a99e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:02:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38096
I20260926 04:02:18.030010 20035 tablet_replica.cc:330] T 17074549f0f94857a75c0aef38b9a99e P e27ef819ceab487f8156060f59151530: stopping tablet replica
I20260926 04:02:18.030114 20035 raft_consensus.cc:2229] T 17074549f0f94857a75c0aef38b9a99e P e27ef819ceab487f8156060f59151530 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 04:02:18.030211 20035 raft_consensus.cc:2258] T 17074549f0f94857a75c0aef38b9a99e P e27ef819ceab487f8156060f59151530 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:02:18.030620 20035 ts_tablet_manager.cc:1905] T 17074549f0f94857a75c0aef38b9a99e P e27ef819ceab487f8156060f59151530: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:02:18.032368 20035 ts_tablet_manager.cc:1918] T 17074549f0f94857a75c0aef38b9a99e P e27ef819ceab487f8156060f59151530: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 04:02:18.032442 20035 log.cc:1196] T 17074549f0f94857a75c0aef38b9a99e P e27ef819ceab487f8156060f59151530: Deleting WAL directory at /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790395095474643-392-0/minicluster-data/ts-2/wal/wals/17074549f0f94857a75c0aef38b9a99e
I20260926 04:02:18.032768 20035 ts_tablet_manager.cc:1939] T 17074549f0f94857a75c0aef38b9a99e P e27ef819ceab487f8156060f59151530: Deleting consensus metadata
I20260926 04:02:18.032827 20033 tablet_replica.cc:330] T 17074549f0f94857a75c0aef38b9a99e P a5e1cc8ae26c49349bde05b594170056: stopping tablet replica
I20260926 04:02:18.033174 19521 catalog_manager.cc:4841] TS e27ef819ceab487f8156060f59151530 (127.0.98.3:42555): tablet 17074549f0f94857a75c0aef38b9a99e (table test-table [id=0b61a65cb8f14327930340b13b0b350b]) successfully deleted
I20260926 04:02:18.035617 20033 raft_consensus.cc:2229] T 17074549f0f94857a75c0aef38b9a99e P a5e1cc8ae26c49349bde05b594170056 [term 1 LEADER]: Raft consensus shutting down.
I20260926 04:02:18.035792 20033 raft_consensus.cc:2258] T 17074549f0f94857a75c0aef38b9a99e P a5e1cc8ae26c49349bde05b594170056 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:02:18.036288 20033 ts_tablet_manager.cc:1905] T 17074549f0f94857a75c0aef38b9a99e P a5e1cc8ae26c49349bde05b594170056: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 04:02:18.036325 19522 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45723 (error 108)
I20260926 04:02:18.036564 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 19731
I20260926 04:02:18.041643 20006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:18.041497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45723 (local address 127.0.0.1:43304)
0926 04:02:18.041548 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:18.041596 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260926 04:02:18.041756 20005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:18.041656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44217 (local address 127.0.0.1:39944)
0926 04:02:18.041710 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:18.041736 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:44217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 04:02:18.041834 19522 catalog_manager.cc:4584] TS 5140e4bba0e5414b9907099e640cf329 (127.0.98.1:45723): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 17074549f0f94857a75c0aef38b9a99e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45723: connect: Connection refused (error 111)
I20260926 04:02:18.041951 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 19865
I20260926 04:02:18.046267 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 19503
2026-09-26T04:02:18Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1930 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-26T04:02:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:02:18Z Disabled control of system clock
I20260926 04:02:18.074021 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395095474643-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395095474643-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395095474643-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41133
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:46129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41133
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 04:02:18.155054 20040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:02:18.155278 20040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:02:18.155298 20040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:02:18.156373 20040 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:02:18.156405 20040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:02:18.156418 20040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:02:18.156430 20040 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:02:18.156442 20040 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:02:18.157807 20040 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395095474643-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395095474643-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41133
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20040
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395095474643-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:02:18.158027 20040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:02:18.158273 20040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:02:18.163012 20048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:02:18.163089 20045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:02:18.163225 20046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:02:18.163123 20040 server_base.cc:1034] running on GCE node
I20260926 04:02:18.163569 20040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:02:18.163820 20040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:02:18.164953 20040 hybrid_clock.cc:648] HybridClock initialized: now 1790395338164938 us; error 27 us; skew 500 ppm
I20260926 04:02:18.166100 20040 webserver.cc:460] Webserver started at http://127.0.98.62:42055/ using document root <none> and password file <none>
I20260926 04:02:18.166320 20040 fs_manager.cc:359] Metadata directory not provided
I20260926 04:02:18.166361 20040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:02:18.166446 20040 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:02:18.167455 20040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395095474643-392-0/minicluster-data/master-0/data/instance:
uuid: "5e5b1245d0e84a82904aeff063d23e03"
format_stamp: "Formatted at 2026-09-26 04:02:18 on dist-test-slave-nhpw"
server_key: "02bf41f5e409919f74d5473873d6a6e8"
server_key_iv: "d787325ad3d2de84e5fcb141a71f59d5"
server_key_version: "encryptionkey@0"
I20260926 04:02:18.167794 20040 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395095474643-392-0/minicluster-data/master-0/wal/instance:
uuid: "5e5b1245d0e84a82904aeff063d23e03"
format_stamp: "Formatted at 2026-09-26 04:02:18 on dist-test-slave-nhpw"
server_key: "02bf41f5e409919f74d5473873d6a6e8"
server_key_iv: "d787325ad3d2de84e5fcb141a71f59d5"
server_key_version: "encryptionkey@0"
I20260926 04:02:18.169221 20040 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 04:02:18.169936 20054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:18.170118 20040 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 04:02:18.170225 20040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395095474643-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395095474643-392-0/minicluster-data/master-0/wal
uuid: "5e5b1245d0e84a82904aeff063d23e03"
format_stamp: "Formatted at 2026-09-26 04:02:18 on dist-test-slave-nhpw"
server_key: "02bf41f5e409919f74d5473873d6a6e8"
server_key_iv: "d787325ad3d2de84e5fcb141a71f59d5"
server_key_version: "encryptionkey@0"
I20260926 04:02:18.170301 20040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395095474643-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395095474643-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395095474643-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:02:18.178944 20040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:02:18.179246 20040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:02:18.179394 20040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:02:18.183307 20040 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41133
I20260926 04:02:18.183339 20106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41133 every 8 connection(s)
I20260926 04:02:18.183672 20040 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
I20260926 04:02:18.184584 20107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:02:18.186811 20107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e5b1245d0e84a82904aeff063d23e03: Bootstrap starting.
I20260926 04:02:18.187314 20107 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e5b1245d0e84a82904aeff063d23e03: Neither blocks nor log segments found. Creating new log.
I20260926 04:02:18.187553 20107 log.cc:824] T 00000000000000000000000000000000 P 5e5b1245d0e84a82904aeff063d23e03: Log is configured to *not* fsync() on all Append() calls
I20260926 04:02:18.188164 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 20040
I20260926 04:02:18.188206 20107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e5b1245d0e84a82904aeff063d23e03: No bootstrap required, opened a new log
I20260926 04:02:18.188282 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395095474643-392-0/minicluster-data/master-0/wal/instance
I20260926 04:02:18.188482 392 external_mini_cluster.cc:1428] Setting key 28956bdfce23bbb55eff6d1259fc8cc2
I20260926 04:02:18.190147 20107 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e5b1245d0e84a82904aeff063d23e03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e5b1245d0e84a82904aeff063d23e03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41133 } }
I20260926 04:02:18.190275 20107 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e5b1245d0e84a82904aeff063d23e03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:02:18.190297 20107 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e5b1245d0e84a82904aeff063d23e03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e5b1245d0e84a82904aeff063d23e03, State: Initialized, Role: FOLLOWER
I20260926 04:02:18.190428 20107 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e5b1245d0e84a82904aeff063d23e03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e5b1245d0e84a82904aeff063d23e03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41133 } }
I20260926 04:02:18.190531 20107 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e5b1245d0e84a82904aeff063d23e03 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:02:18.190572 20107 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e5b1245d0e84a82904aeff063d23e03 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:02:18.190615 20107 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e5b1245d0e84a82904aeff063d23e03 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:02:18.191126 20107 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e5b1245d0e84a82904aeff063d23e03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e5b1245d0e84a82904aeff063d23e03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41133 } }
I20260926 04:02:18.191242 20107 leader_election.cc:302] T 00000000000000000000000000000000 P 5e5b1245d0e84a82904aeff063d23e03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e5b1245d0e84a82904aeff063d23e03; no voters:
I20260926 04:02:18.191428 20107 leader_election.cc:288] T 00000000000000000000000000000000 P 5e5b1245d0e84a82904aeff063d23e03 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 04:02:18.191464 20112 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e5b1245d0e84a82904aeff063d23e03 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:02:18.191602 20112 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e5b1245d0e84a82904aeff063d23e03 [term 1 LEADER]: Becoming Leader. State: Replica: 5e5b1245d0e84a82904aeff063d23e03, State: Running, Role: LEADER
I20260926 04:02:18.191685 20107 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e5b1245d0e84a82904aeff063d23e03 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:02:18.191685 20112 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e5b1245d0e84a82904aeff063d23e03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e5b1245d0e84a82904aeff063d23e03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41133 } }
I20260926 04:02:18.192075 20114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e5b1245d0e84a82904aeff063d23e03 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e5b1245d0e84a82904aeff063d23e03. Latest consensus state: current_term: 1 leader_uuid: "5e5b1245d0e84a82904aeff063d23e03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e5b1245d0e84a82904aeff063d23e03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41133 } } }
I20260926 04:02:18.192268 20114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e5b1245d0e84a82904aeff063d23e03 [sys.catalog]: This master's current role is: LEADER
I20260926 04:02:18.192073 20113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e5b1245d0e84a82904aeff063d23e03 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e5b1245d0e84a82904aeff063d23e03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e5b1245d0e84a82904aeff063d23e03" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41133 } } }
I20260926 04:02:18.192571 20113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e5b1245d0e84a82904aeff063d23e03 [sys.catalog]: This master's current role is: LEADER
W20260926 04:02:18.193461 20128 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5e5b1245d0e84a82904aeff063d23e03: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 04:02:18.193518 20128 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 04:02:18.193565 20121 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:02:18.193742 20121 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:02:18.195376 20121 catalog_manager.cc:1347] Generated new cluster ID: ea9e11ab5eda4c0296f85874bdc46669
I20260926 04:02:18.195436 20121 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 04:02:18.195629 20111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:18.189076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46942 (local address 127.0.98.62:41133)
0926 04:02:18.189273 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:18.189288 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:18.189312 (+ 24us) server_negotiation.cc:407] Connection header received
0926 04:02:18.190139 (+ 827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:18.190147 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:18.190209 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:18.190385 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:18.190642 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:18.192287 (+ 1645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:18.193334 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:18.193884 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:18.194025 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:18.194724 (+ 699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:18.194729 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:18.194729 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:02:18.194937 (+ 208us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:18.194969 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:02:18.194981 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:18.195074 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:18.195157 (+ 83us) server_negotiation.cc:299] Negotiation successful
0926 04:02:18.195333 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":97,"threads_started":1}
I20260926 04:02:18.214519 20121 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:02:18.215198 20121 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:02:18.223443 20121 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e5b1245d0e84a82904aeff063d23e03: Generated new TSK 0
I20260926 04:02:18.223628 20121 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 04:02:18.233606 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395095474643-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395095474643-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395095474643-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41133
--builtin_ntp_servers=127.0.98.20:46129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 04:02:18.317095 20131 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:02:18.317324 20131 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:02:18.317358 20131 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:02:18.318785 20131 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:02:18.318869 20131 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:02:18.318918 20131 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260926 04:02:18.320369 20131 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395095474643-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395095474643-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20131
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395095474643-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:02:18.320762 20131 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:02:18.321121 20131 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:02:18.326746 20136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:02:18.326792 20139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:02:18.326798 20137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:02:18.327112 20131 server_base.cc:1034] running on GCE node
I20260926 04:02:18.327301 20131 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:02:18.327534 20131 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:02:18.328699 20131 hybrid_clock.cc:648] HybridClock initialized: now 1790395338328684 us; error 31 us; skew 500 ppm
I20260926 04:02:18.329873 20131 webserver.cc:460] Webserver started at http://127.0.98.1:41439/ using document root <none> and password file <none>
I20260926 04:02:18.330075 20131 fs_manager.cc:359] Metadata directory not provided
I20260926 04:02:18.330116 20131 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:02:18.330202 20131 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:02:18.331152 20131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395095474643-392-0/minicluster-data/ts-0/data/instance:
uuid: "cee54717250642078a55edf53008319f"
format_stamp: "Formatted at 2026-09-26 04:02:18 on dist-test-slave-nhpw"
server_key: "ca42c4f70d1a5ef335328cb1b8494ed4"
server_key_iv: "f18da72fe8f790ca4fb031e75a0532c1"
server_key_version: "encryptionkey@0"
I20260926 04:02:18.331442 20131 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395095474643-392-0/minicluster-data/ts-0/wal/instance:
uuid: "cee54717250642078a55edf53008319f"
format_stamp: "Formatted at 2026-09-26 04:02:18 on dist-test-slave-nhpw"
server_key: "ca42c4f70d1a5ef335328cb1b8494ed4"
server_key_iv: "f18da72fe8f790ca4fb031e75a0532c1"
server_key_version: "encryptionkey@0"
I20260926 04:02:18.332626 20131 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 04:02:18.333361 20145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:18.333557 20131 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 04:02:18.333659 20131 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395095474643-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395095474643-392-0/minicluster-data/ts-0/wal
uuid: "cee54717250642078a55edf53008319f"
format_stamp: "Formatted at 2026-09-26 04:02:18 on dist-test-slave-nhpw"
server_key: "ca42c4f70d1a5ef335328cb1b8494ed4"
server_key_iv: "f18da72fe8f790ca4fb031e75a0532c1"
server_key_version: "encryptionkey@0"
I20260926 04:02:18.333751 20131 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395095474643-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395095474643-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395095474643-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:02:18.359048 20131 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:02:18.359434 20131 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:02:18.359587 20131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:02:18.359793 20131 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:02:18.360121 20131 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:02:18.360154 20131 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:18.360176 20131 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:02:18.360224 20131 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:18.366176 20131 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45941
I20260926 04:02:18.366216 20258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45941 every 8 connection(s)
I20260926 04:02:18.366593 20131 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395095474643-392-0/minicluster-data/ts-0/data/info.pb
I20260926 04:02:18.368860 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 20131
I20260926 04:02:18.368979 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395095474643-392-0/minicluster-data/ts-0/wal/instance
I20260926 04:02:18.369189 392 external_mini_cluster.cc:1428] Setting key e068eedd273074d91f18a69b926364fe
I20260926 04:02:18.370252 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395095474643-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395095474643-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395095474643-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41133
--builtin_ntp_servers=127.0.98.20:46129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 04:02:18.373047 20111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:18.367346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55487 (local address 127.0.98.62:41133)
0926 04:02:18.367416 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:18.367418 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:18.367617 (+ 199us) server_negotiation.cc:407] Connection header received
0926 04:02:18.368103 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:18.368105 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:18.368117 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:18.368185 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:18.369150 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:18.369795 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:18.371305 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:18.371583 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:18.371664 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:18.372712 (+ 1048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:18.372714 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:18.372714 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:02:18.372718 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:18.372733 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:02:18.372736 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:18.372784 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:18.372893 (+ 109us) server_negotiation.cc:299] Negotiation successful
0926 04:02:18.372922 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 04:02:18.373255 20261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:18.367295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41133 (local address 127.0.98.1:55487)
0926 04:02:18.367485 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:18.367525 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 04:02:18.368005 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:02:18.368285 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:02:18.368289 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:02:18.368297 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:02:18.368997 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:18.369004 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:18.369922 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:18.369923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:18.371224 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:18.371227 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:18.371680 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:18.371681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:18.371787 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:18.372529 (+ 742us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 04:02:18.372530 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:02:18.372558 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 04:02:18.372581 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:02:18.372803 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:02:18.372808 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:02:18.372812 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 04:02:18.372885 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 04:02:18.372999 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":88,"threads_started":1}
I20260926 04:02:18.373729 20259 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41133
I20260926 04:02:18.373813 20259 heartbeater.cc:464] Registering TS with master...
I20260926 04:02:18.374068 20259 heartbeater.cc:511] Master 127.0.98.62:41133 requested a full tablet report, sending...
I20260926 04:02:18.374724 20071 ts_manager.cc:194] Registered new tserver with Master: cee54717250642078a55edf53008319f (127.0.98.1:45941)
I20260926 04:02:18.375749 20071 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:55487
W20260926 04:02:18.450714 20262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:02:18.450906 20262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:02:18.450927 20262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:02:18.451992 20262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:02:18.452025 20262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:02:18.452050 20262 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260926 04:02:18.453344 20262 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395095474643-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395095474643-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395095474643-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:02:18.453529 20262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:02:18.453780 20262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:02:18.458683 20267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:02:18.458694 20268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:02:18.458779 20262 server_base.cc:1034] running on GCE node
W20260926 04:02:18.458701 20270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:02:18.459066 20262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:02:18.459328 20262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:02:18.460505 20262 hybrid_clock.cc:648] HybridClock initialized: now 1790395338460483 us; error 31 us; skew 500 ppm
I20260926 04:02:18.461743 20262 webserver.cc:460] Webserver started at http://127.0.98.2:36559/ using document root <none> and password file <none>
I20260926 04:02:18.462073 20262 fs_manager.cc:359] Metadata directory not provided
I20260926 04:02:18.462144 20262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:02:18.462278 20262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:02:18.463310 20262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395095474643-392-0/minicluster-data/ts-1/data/instance:
uuid: "b5de051fb2eb4725ba02a4ce070d1514"
format_stamp: "Formatted at 2026-09-26 04:02:18 on dist-test-slave-nhpw"
server_key: "d30c36af5847d885e27945cf18a12e33"
server_key_iv: "401d4686ede36279728f882cccde27f4"
server_key_version: "encryptionkey@0"
I20260926 04:02:18.463675 20262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395095474643-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b5de051fb2eb4725ba02a4ce070d1514"
format_stamp: "Formatted at 2026-09-26 04:02:18 on dist-test-slave-nhpw"
server_key: "d30c36af5847d885e27945cf18a12e33"
server_key_iv: "401d4686ede36279728f882cccde27f4"
server_key_version: "encryptionkey@0"
I20260926 04:02:18.465044 20262 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 04:02:18.465813 20276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:18.465988 20262 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 04:02:18.466094 20262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395095474643-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395095474643-392-0/minicluster-data/ts-1/wal
uuid: "b5de051fb2eb4725ba02a4ce070d1514"
format_stamp: "Formatted at 2026-09-26 04:02:18 on dist-test-slave-nhpw"
server_key: "d30c36af5847d885e27945cf18a12e33"
server_key_iv: "401d4686ede36279728f882cccde27f4"
server_key_version: "encryptionkey@0"
I20260926 04:02:18.466176 20262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395095474643-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395095474643-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395095474643-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:02:18.494343 20262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:02:18.494699 20262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:02:18.494858 20262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:02:18.495108 20262 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:02:18.495487 20262 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:02:18.495548 20262 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:18.495591 20262 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:02:18.495621 20262 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:18.501446 20262 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36505
I20260926 04:02:18.501502 20389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36505 every 8 connection(s)
I20260926 04:02:18.501873 20262 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395095474643-392-0/minicluster-data/ts-1/data/info.pb
I20260926 04:02:18.506304 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 20262
I20260926 04:02:18.506394 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395095474643-392-0/minicluster-data/ts-1/wal/instance
I20260926 04:02:18.506629 392 external_mini_cluster.cc:1428] Setting key f9261c85726df2afc8536fe5328b0419
I20260926 04:02:18.507551 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395095474643-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395095474643-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395095474643-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41133
--builtin_ntp_servers=127.0.98.20:46129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 04:02:18.508967 20111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:18.502646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60233 (local address 127.0.98.62:41133)
0926 04:02:18.502710 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:18.502712 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:18.502954 (+ 242us) server_negotiation.cc:407] Connection header received
0926 04:02:18.503502 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:18.503503 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:18.503516 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:18.503570 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:18.504262 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:18.504783 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:18.505772 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:18.505975 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:18.506070 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:18.507206 (+ 1136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:18.507208 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:18.507208 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:02:18.507214 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:18.507232 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:02:18.507236 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:18.507295 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:18.508825 (+ 1530us) server_negotiation.cc:299] Negotiation successful
0926 04:02:18.508859 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 04:02:18.509078 20392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:18.502628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41133 (local address 127.0.98.2:60233)
0926 04:02:18.502832 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:18.502865 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 04:02:18.503385 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:02:18.503650 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:02:18.503654 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:02:18.503665 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:02:18.504165 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:18.504169 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:18.504880 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:18.504881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:18.505679 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:18.505681 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:18.506080 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:18.506081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:18.506230 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:18.507045 (+ 815us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 04:02:18.507046 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:02:18.507074 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 04:02:18.507096 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:02:18.508710 (+ 1614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:02:18.508718 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:02:18.508721 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 04:02:18.508810 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 04:02:18.508895 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":86,"threads_started":1}
I20260926 04:02:18.509630 20390 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41133
I20260926 04:02:18.509727 20390 heartbeater.cc:464] Registering TS with master...
I20260926 04:02:18.509963 20390 heartbeater.cc:511] Master 127.0.98.62:41133 requested a full tablet report, sending...
I20260926 04:02:18.510381 20071 ts_manager.cc:194] Registered new tserver with Master: b5de051fb2eb4725ba02a4ce070d1514 (127.0.98.2:36505)
I20260926 04:02:18.510903 20071 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:60233
W20260926 04:02:18.588742 20393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:02:18.588943 20393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:02:18.588963 20393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:02:18.590121 20393 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:02:18.590162 20393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:02:18.590188 20393 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260926 04:02:18.591521 20393 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395095474643-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395095474643-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41133
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20393
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395095474643-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:02:18.591810 20393 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:02:18.592104 20393 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:02:18.597211 20398 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:02:18.597232 20401 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:02:18.597211 20399 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:02:18.597330 20393 server_base.cc:1034] running on GCE node
I20260926 04:02:18.597663 20393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:02:18.597911 20393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:02:18.599087 20393 hybrid_clock.cc:648] HybridClock initialized: now 1790395338599091 us; error 50 us; skew 500 ppm
I20260926 04:02:18.600414 20393 webserver.cc:460] Webserver started at http://127.0.98.3:39501/ using document root <none> and password file <none>
I20260926 04:02:18.600711 20393 fs_manager.cc:359] Metadata directory not provided
I20260926 04:02:18.600783 20393 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:02:18.600943 20393 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:02:18.602042 20393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395095474643-392-0/minicluster-data/ts-2/data/instance:
uuid: "e09a6f57e1b64b19b2ade354e28fa698"
format_stamp: "Formatted at 2026-09-26 04:02:18 on dist-test-slave-nhpw"
server_key: "ebfb7e6d38f1d65a871b1b95b93a509b"
server_key_iv: "d40a4733e66d6c497953aab8e2669d37"
server_key_version: "encryptionkey@0"
I20260926 04:02:18.602437 20393 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395095474643-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e09a6f57e1b64b19b2ade354e28fa698"
format_stamp: "Formatted at 2026-09-26 04:02:18 on dist-test-slave-nhpw"
server_key: "ebfb7e6d38f1d65a871b1b95b93a509b"
server_key_iv: "d40a4733e66d6c497953aab8e2669d37"
server_key_version: "encryptionkey@0"
I20260926 04:02:18.603742 20393 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 04:02:18.604477 20407 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:18.604666 20393 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 04:02:18.604774 20393 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395095474643-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395095474643-392-0/minicluster-data/ts-2/wal
uuid: "e09a6f57e1b64b19b2ade354e28fa698"
format_stamp: "Formatted at 2026-09-26 04:02:18 on dist-test-slave-nhpw"
server_key: "ebfb7e6d38f1d65a871b1b95b93a509b"
server_key_iv: "d40a4733e66d6c497953aab8e2669d37"
server_key_version: "encryptionkey@0"
I20260926 04:02:18.604862 20393 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395095474643-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395095474643-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395095474643-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:02:18.620306 20393 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:02:18.620666 20393 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:02:18.620828 20393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:02:18.621065 20393 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 04:02:18.621411 20393 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 04:02:18.621467 20393 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:18.621502 20393 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 04:02:18.621532 20393 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:18.628849 20393 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43761
I20260926 04:02:18.628984 20520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43761 every 8 connection(s)
I20260926 04:02:18.629339 20393 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395095474643-392-0/minicluster-data/ts-2/data/info.pb
I20260926 04:02:18.632849 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 20393
I20260926 04:02:18.632968 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395095474643-392-0/minicluster-data/ts-2/wal/instance
I20260926 04:02:18.633188 392 external_mini_cluster.cc:1428] Setting key c1d1544712dbfc70ad3131bf93107ab1
I20260926 04:02:18.636195 20111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:18.630918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48377 (local address 127.0.98.62:41133)
0926 04:02:18.630964 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:18.630967 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:18.631282 (+ 315us) server_negotiation.cc:407] Connection header received
0926 04:02:18.632015 (+ 733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:18.632017 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:18.632033 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:18.632116 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:18.633085 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:18.633813 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:18.634797 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:18.635008 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:18.635083 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:18.635942 (+ 859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:18.635944 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:18.635945 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:02:18.635950 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:18.635965 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:02:18.635969 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:18.636025 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:18.636092 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 04:02:18.636120 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 04:02:18.636405 20523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:18.630938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41133 (local address 127.0.98.3:48377)
0926 04:02:18.631158 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:18.631206 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 04:02:18.631881 (+ 675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:02:18.632192 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:02:18.632196 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:02:18.632206 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:02:18.632997 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:18.633004 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:18.633888 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:18.633889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:18.634673 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:18.634676 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:18.635088 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:18.635089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:18.635233 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:18.635782 (+ 549us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 04:02:18.635782 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:02:18.635810 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 04:02:18.635830 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:02:18.636023 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:02:18.636029 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:02:18.636032 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 04:02:18.636097 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 04:02:18.636194 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":102,"threads_started":1}
I20260926 04:02:18.636899 20521 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41133
I20260926 04:02:18.636973 20521 heartbeater.cc:464] Registering TS with master...
I20260926 04:02:18.637181 20521 heartbeater.cc:511] Master 127.0.98.62:41133 requested a full tablet report, sending...
I20260926 04:02:18.637530 20071 ts_manager.cc:194] Registered new tserver with Master: e09a6f57e1b64b19b2ade354e28fa698 (127.0.98.3:43761)
I20260926 04:02:18.637897 20071 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:48377
I20260926 04:02:18.644932 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 04:02:18.649353 20111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:18.646341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46950 (local address 127.0.98.62:41133)
0926 04:02:18.646398 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:18.646400 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:18.646549 (+ 149us) server_negotiation.cc:407] Connection header received
0926 04:02:18.646626 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:18.646628 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:18.646646 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:18.646717 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:18.646897 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:18.647389 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:18.648149 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:18.648336 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:18.648432 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:18.649084 (+ 652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:18.649086 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:18.649086 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:02:18.649091 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:18.649103 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:02:18.649107 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:18.649183 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:18.649245 (+ 62us) server_negotiation.cc:299] Negotiation successful
0926 04:02:18.649268 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 04:02:18.650686 20071 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46950:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 04:02:18.652166 20071 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 04:02:18.658536 20529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:18.654341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45941 (local address 127.0.0.1:32810)
0926 04:02:18.654523 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:18.654564 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 04:02:18.654626 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:02:18.654938 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:02:18.654941 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:02:18.654944 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:02:18.655011 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:18.655013 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:18.655974 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:18.655975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:18.657305 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:18.657307 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:18.658237 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:18.658239 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:18.658322 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:18.658336 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 04:02:18.658414 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 04:02:18.658464 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":54,"thread_start_us":95,"threads_started":1}
I20260926 04:02:18.658725 20531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:18.654456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32810 (local address 127.0.98.1:45941)
0926 04:02:18.654739 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:18.654742 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:18.654753 (+ 11us) server_negotiation.cc:407] Connection header received
0926 04:02:18.654795 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:18.654797 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:18.654812 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:18.654890 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:02:18.655111 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:18.655879 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:18.657404 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:18.658143 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:18.658227 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:18.658324 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:18.658482 (+ 158us) server_negotiation.cc:299] Negotiation successful
0926 04:02:18.658529 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":96,"threads_started":1}
I20260926 04:02:18.658906 20532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:18.654455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59916 (local address 127.0.98.2:36505)
0926 04:02:18.654663 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:18.654668 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:18.654781 (+ 113us) server_negotiation.cc:407] Connection header received
0926 04:02:18.654820 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:18.654822 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:18.654839 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:18.654902 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:02:18.655166 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:18.655891 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:18.657443 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:18.658208 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:18.658267 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:18.658337 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:18.658778 (+ 441us) server_negotiation.cc:299] Negotiation successful
0926 04:02:18.658825 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":99,"threads_started":1}
I20260926 04:02:18.659056 20530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:18.654341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36505 (local address 127.0.0.1:59916)
0926 04:02:18.654647 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:18.654658 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 04:02:18.654721 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:02:18.655035 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:02:18.655036 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:02:18.655038 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:02:18.655085 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:18.655086 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:18.655982 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:18.655984 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:18.657366 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:18.657369 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:18.658371 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:18.658372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:18.658505 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:18.658510 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 04:02:18.658596 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 04:02:18.658984 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"mutex_wait_us":48,"thread_start_us":66,"threads_started":1}
I20260926 04:02:18.659540 20193 tablet_service.cc:1467] Processing CreateTablet for tablet efba85a7e14341de92f89b81cb1fc88d (DEFAULT_TABLE table=test-table [id=d659155afae4463fae0c68ff5a0abc44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:02:18.659708 20534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:18.654737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43786 (local address 127.0.98.3:43761)
0926 04:02:18.654941 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:18.654945 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:18.656109 (+ 1164us) server_negotiation.cc:407] Connection header received
0926 04:02:18.656156 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:18.656158 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:18.656174 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:18.656256 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:02:18.656362 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:18.657456 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:18.658330 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:18.659368 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:18.659474 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:18.659564 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:18.659596 (+ 32us) server_negotiation.cc:299] Negotiation successful
0926 04:02:18.659632 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":99,"threads_started":1}
I20260926 04:02:18.659675 20324 tablet_service.cc:1467] Processing CreateTablet for tablet efba85a7e14341de92f89b81cb1fc88d (DEFAULT_TABLE table=test-table [id=d659155afae4463fae0c68ff5a0abc44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:02:18.659690 20533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:18.654538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43761 (local address 127.0.0.1:43786)
0926 04:02:18.656026 (+ 1488us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:18.656036 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 04:02:18.656089 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:02:18.656264 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:02:18.656265 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:02:18.656268 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:02:18.656303 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:18.656304 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:18.657522 (+ 1218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:18.657523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:18.658234 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:18.658235 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:18.659477 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:18.659479 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:18.659536 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:18.659540 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 04:02:18.659586 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 04:02:18.659622 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1470,"thread_start_us":72,"threads_started":1}
I20260926 04:02:18.659929 20193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efba85a7e14341de92f89b81cb1fc88d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:02:18.659967 20324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efba85a7e14341de92f89b81cb1fc88d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:02:18.660450 20448 tablet_service.cc:1467] Processing CreateTablet for tablet efba85a7e14341de92f89b81cb1fc88d (DEFAULT_TABLE table=test-table [id=d659155afae4463fae0c68ff5a0abc44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 04:02:18.660787 20448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efba85a7e14341de92f89b81cb1fc88d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:02:18.663240 20535 tablet_bootstrap.cc:492] T efba85a7e14341de92f89b81cb1fc88d P e09a6f57e1b64b19b2ade354e28fa698: Bootstrap starting.
I20260926 04:02:18.663575 20537 tablet_bootstrap.cc:492] T efba85a7e14341de92f89b81cb1fc88d P b5de051fb2eb4725ba02a4ce070d1514: Bootstrap starting.
I20260926 04:02:18.663950 20535 tablet_bootstrap.cc:654] T efba85a7e14341de92f89b81cb1fc88d P e09a6f57e1b64b19b2ade354e28fa698: Neither blocks nor log segments found. Creating new log.
I20260926 04:02:18.664237 20537 tablet_bootstrap.cc:654] T efba85a7e14341de92f89b81cb1fc88d P b5de051fb2eb4725ba02a4ce070d1514: Neither blocks nor log segments found. Creating new log.
I20260926 04:02:18.664247 20535 log.cc:824] T efba85a7e14341de92f89b81cb1fc88d P e09a6f57e1b64b19b2ade354e28fa698: Log is configured to *not* fsync() on all Append() calls
I20260926 04:02:18.664587 20537 log.cc:824] T efba85a7e14341de92f89b81cb1fc88d P b5de051fb2eb4725ba02a4ce070d1514: Log is configured to *not* fsync() on all Append() calls
I20260926 04:02:18.664803 20536 tablet_bootstrap.cc:492] T efba85a7e14341de92f89b81cb1fc88d P cee54717250642078a55edf53008319f: Bootstrap starting.
I20260926 04:02:18.665298 20537 tablet_bootstrap.cc:492] T efba85a7e14341de92f89b81cb1fc88d P b5de051fb2eb4725ba02a4ce070d1514: No bootstrap required, opened a new log
I20260926 04:02:18.665351 20537 ts_tablet_manager.cc:1397] T efba85a7e14341de92f89b81cb1fc88d P b5de051fb2eb4725ba02a4ce070d1514: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 04:02:18.665393 20536 tablet_bootstrap.cc:654] T efba85a7e14341de92f89b81cb1fc88d P cee54717250642078a55edf53008319f: Neither blocks nor log segments found. Creating new log.
I20260926 04:02:18.665663 20536 log.cc:824] T efba85a7e14341de92f89b81cb1fc88d P cee54717250642078a55edf53008319f: Log is configured to *not* fsync() on all Append() calls
I20260926 04:02:18.666198 20535 tablet_bootstrap.cc:492] T efba85a7e14341de92f89b81cb1fc88d P e09a6f57e1b64b19b2ade354e28fa698: No bootstrap required, opened a new log
I20260926 04:02:18.666271 20535 ts_tablet_manager.cc:1397] T efba85a7e14341de92f89b81cb1fc88d P e09a6f57e1b64b19b2ade354e28fa698: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 04:02:18.666462 20536 tablet_bootstrap.cc:492] T efba85a7e14341de92f89b81cb1fc88d P cee54717250642078a55edf53008319f: No bootstrap required, opened a new log
I20260926 04:02:18.666522 20536 ts_tablet_manager.cc:1397] T efba85a7e14341de92f89b81cb1fc88d P cee54717250642078a55edf53008319f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 04:02:18.667454 20537 raft_consensus.cc:348] T efba85a7e14341de92f89b81cb1fc88d P b5de051fb2eb4725ba02a4ce070d1514 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee54717250642078a55edf53008319f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45941 } } peers { permanent_uuid: "e09a6f57e1b64b19b2ade354e28fa698" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43761 } } peers { permanent_uuid: "b5de051fb2eb4725ba02a4ce070d1514" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36505 } }
I20260926 04:02:18.667639 20537 raft_consensus.cc:374] T efba85a7e14341de92f89b81cb1fc88d P b5de051fb2eb4725ba02a4ce070d1514 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:02:18.667685 20537 raft_consensus.cc:729] T efba85a7e14341de92f89b81cb1fc88d P b5de051fb2eb4725ba02a4ce070d1514 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5de051fb2eb4725ba02a4ce070d1514, State: Initialized, Role: FOLLOWER
I20260926 04:02:18.667799 20537 consensus_queue.cc:260] T efba85a7e14341de92f89b81cb1fc88d P b5de051fb2eb4725ba02a4ce070d1514 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee54717250642078a55edf53008319f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45941 } } peers { permanent_uuid: "e09a6f57e1b64b19b2ade354e28fa698" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43761 } } peers { permanent_uuid: "b5de051fb2eb4725ba02a4ce070d1514" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36505 } }
I20260926 04:02:18.668085 20537 ts_tablet_manager.cc:1428] T efba85a7e14341de92f89b81cb1fc88d P b5de051fb2eb4725ba02a4ce070d1514: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 04:02:18.668120 20390 heartbeater.cc:503] Master 127.0.98.62:41133 was elected leader, sending a full tablet report...
I20260926 04:02:18.669087 20535 raft_consensus.cc:348] T efba85a7e14341de92f89b81cb1fc88d P e09a6f57e1b64b19b2ade354e28fa698 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee54717250642078a55edf53008319f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45941 } } peers { permanent_uuid: "e09a6f57e1b64b19b2ade354e28fa698" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43761 } } peers { permanent_uuid: "b5de051fb2eb4725ba02a4ce070d1514" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36505 } }
I20260926 04:02:18.669155 20536 raft_consensus.cc:348] T efba85a7e14341de92f89b81cb1fc88d P cee54717250642078a55edf53008319f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee54717250642078a55edf53008319f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45941 } } peers { permanent_uuid: "e09a6f57e1b64b19b2ade354e28fa698" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43761 } } peers { permanent_uuid: "b5de051fb2eb4725ba02a4ce070d1514" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36505 } }
I20260926 04:02:18.669260 20535 raft_consensus.cc:374] T efba85a7e14341de92f89b81cb1fc88d P e09a6f57e1b64b19b2ade354e28fa698 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:02:18.669274 20536 raft_consensus.cc:374] T efba85a7e14341de92f89b81cb1fc88d P cee54717250642078a55edf53008319f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:02:18.669297 20535 raft_consensus.cc:729] T efba85a7e14341de92f89b81cb1fc88d P e09a6f57e1b64b19b2ade354e28fa698 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e09a6f57e1b64b19b2ade354e28fa698, State: Initialized, Role: FOLLOWER
I20260926 04:02:18.669307 20536 raft_consensus.cc:729] T efba85a7e14341de92f89b81cb1fc88d P cee54717250642078a55edf53008319f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cee54717250642078a55edf53008319f, State: Initialized, Role: FOLLOWER
I20260926 04:02:18.669426 20536 consensus_queue.cc:260] T efba85a7e14341de92f89b81cb1fc88d P cee54717250642078a55edf53008319f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee54717250642078a55edf53008319f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45941 } } peers { permanent_uuid: "e09a6f57e1b64b19b2ade354e28fa698" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43761 } } peers { permanent_uuid: "b5de051fb2eb4725ba02a4ce070d1514" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36505 } }
I20260926 04:02:18.669435 20535 consensus_queue.cc:260] T efba85a7e14341de92f89b81cb1fc88d P e09a6f57e1b64b19b2ade354e28fa698 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee54717250642078a55edf53008319f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45941 } } peers { permanent_uuid: "e09a6f57e1b64b19b2ade354e28fa698" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43761 } } peers { permanent_uuid: "b5de051fb2eb4725ba02a4ce070d1514" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36505 } }
I20260926 04:02:18.669695 20535 ts_tablet_manager.cc:1428] T efba85a7e14341de92f89b81cb1fc88d P e09a6f57e1b64b19b2ade354e28fa698: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260926 04:02:18.669694 20536 ts_tablet_manager.cc:1428] T efba85a7e14341de92f89b81cb1fc88d P cee54717250642078a55edf53008319f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 04:02:18.669759 20521 heartbeater.cc:503] Master 127.0.98.62:41133 was elected leader, sending a full tablet report...
I20260926 04:02:18.669819 20259 heartbeater.cc:503] Master 127.0.98.62:41133 was elected leader, sending a full tablet report...
I20260926 04:02:18.742045 20543 raft_consensus.cc:482] T efba85a7e14341de92f89b81cb1fc88d P e09a6f57e1b64b19b2ade354e28fa698 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 04:02:18.742237 20543 raft_consensus.cc:504] T efba85a7e14341de92f89b81cb1fc88d P e09a6f57e1b64b19b2ade354e28fa698 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee54717250642078a55edf53008319f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45941 } } peers { permanent_uuid: "e09a6f57e1b64b19b2ade354e28fa698" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43761 } } peers { permanent_uuid: "b5de051fb2eb4725ba02a4ce070d1514" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36505 } }
I20260926 04:02:18.742656 20543 leader_election.cc:288] T efba85a7e14341de92f89b81cb1fc88d P e09a6f57e1b64b19b2ade354e28fa698 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cee54717250642078a55edf53008319f (127.0.98.1:45941), b5de051fb2eb4725ba02a4ce070d1514 (127.0.98.2:36505)
I20260926 04:02:18.745833 20544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:18.742877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45941 (local address 127.0.98.3:37947)
0926 04:02:18.743062 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:18.743071 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 04:02:18.743113 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:02:18.743296 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:02:18.743297 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:02:18.743298 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:02:18.743376 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:18.743378 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:18.744120 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:18.744121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:18.744906 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:18.744908 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:18.745531 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:18.745532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:18.745607 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:18.745613 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 04:02:18.745669 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 04:02:18.745716 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":2,"thread_start_us":103,"threads_started":1}
I20260926 04:02:18.745823 20531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:18.743039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37947 (local address 127.0.98.1:45941)
0926 04:02:18.743090 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:18.743092 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:18.743105 (+ 13us) server_negotiation.cc:407] Connection header received
0926 04:02:18.743169 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:18.743170 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:18.743202 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:18.743286 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:02:18.743475 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:18.744018 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:18.744991 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:18.745450 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:18.745510 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:18.745520 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:18.745686 (+ 166us) server_negotiation.cc:299] Negotiation successful
0926 04:02:18.745724 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 04:02:18.745935 20523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:18.742865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36505 (local address 127.0.98.3:47561)
0926 04:02:18.742928 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:18.742940 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 04:02:18.743020 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:02:18.743227 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:02:18.743228 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:02:18.743230 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 04:02:18.743313 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:18.743315 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:18.743976 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:18.743977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:18.745003 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:18.745005 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:18.745677 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:18.745679 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:18.745750 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:18.745753 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 04:02:18.745816 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 04:02:18.745857 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":1}
I20260926 04:02:18.745970 20532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:18.742966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47561 (local address 127.0.98.2:36505)
0926 04:02:18.743017 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:18.743019 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:18.743033 (+ 14us) server_negotiation.cc:407] Connection header received
0926 04:02:18.743124 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:18.743125 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:18.743141 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:18.743204 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 04:02:18.743407 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:18.743889 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:18.745120 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:18.745567 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:18.745657 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:18.745668 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:18.745838 (+ 170us) server_negotiation.cc:299] Negotiation successful
0926 04:02:18.745889 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 04:02:18.746338 20213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efba85a7e14341de92f89b81cb1fc88d" candidate_uuid: "e09a6f57e1b64b19b2ade354e28fa698" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cee54717250642078a55edf53008319f" is_pre_election: true
I20260926 04:02:18.746393 20344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efba85a7e14341de92f89b81cb1fc88d" candidate_uuid: "e09a6f57e1b64b19b2ade354e28fa698" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5de051fb2eb4725ba02a4ce070d1514" is_pre_election: true
I20260926 04:02:18.746536 20213 raft_consensus.cc:2454] T efba85a7e14341de92f89b81cb1fc88d P cee54717250642078a55edf53008319f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e09a6f57e1b64b19b2ade354e28fa698 in term 0.
I20260926 04:02:18.746551 20344 raft_consensus.cc:2454] T efba85a7e14341de92f89b81cb1fc88d P b5de051fb2eb4725ba02a4ce070d1514 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e09a6f57e1b64b19b2ade354e28fa698 in term 0.
I20260926 04:02:18.746773 20410 leader_election.cc:302] T efba85a7e14341de92f89b81cb1fc88d P e09a6f57e1b64b19b2ade354e28fa698 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cee54717250642078a55edf53008319f, e09a6f57e1b64b19b2ade354e28fa698; no voters:
I20260926 04:02:18.746917 20543 raft_consensus.cc:2781] T efba85a7e14341de92f89b81cb1fc88d P e09a6f57e1b64b19b2ade354e28fa698 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 04:02:18.746986 20543 raft_consensus.cc:482] T efba85a7e14341de92f89b81cb1fc88d P e09a6f57e1b64b19b2ade354e28fa698 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 04:02:18.747005 20543 raft_consensus.cc:3037] T efba85a7e14341de92f89b81cb1fc88d P e09a6f57e1b64b19b2ade354e28fa698 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:02:18.747718 20543 raft_consensus.cc:504] T efba85a7e14341de92f89b81cb1fc88d P e09a6f57e1b64b19b2ade354e28fa698 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee54717250642078a55edf53008319f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45941 } } peers { permanent_uuid: "e09a6f57e1b64b19b2ade354e28fa698" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43761 } } peers { permanent_uuid: "b5de051fb2eb4725ba02a4ce070d1514" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36505 } }
I20260926 04:02:18.747861 20543 leader_election.cc:288] T efba85a7e14341de92f89b81cb1fc88d P e09a6f57e1b64b19b2ade354e28fa698 [CANDIDATE]: Term 1 election: Requested vote from peers cee54717250642078a55edf53008319f (127.0.98.1:45941), b5de051fb2eb4725ba02a4ce070d1514 (127.0.98.2:36505)
I20260926 04:02:18.748039 20213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efba85a7e14341de92f89b81cb1fc88d" candidate_uuid: "e09a6f57e1b64b19b2ade354e28fa698" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cee54717250642078a55edf53008319f"
I20260926 04:02:18.748060 20344 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efba85a7e14341de92f89b81cb1fc88d" candidate_uuid: "e09a6f57e1b64b19b2ade354e28fa698" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5de051fb2eb4725ba02a4ce070d1514"
I20260926 04:02:18.748116 20213 raft_consensus.cc:3037] T efba85a7e14341de92f89b81cb1fc88d P cee54717250642078a55edf53008319f [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:02:18.748184 20344 raft_consensus.cc:3037] T efba85a7e14341de92f89b81cb1fc88d P b5de051fb2eb4725ba02a4ce070d1514 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:02:18.748920 20213 raft_consensus.cc:2454] T efba85a7e14341de92f89b81cb1fc88d P cee54717250642078a55edf53008319f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e09a6f57e1b64b19b2ade354e28fa698 in term 1.
I20260926 04:02:18.749109 20344 raft_consensus.cc:2454] T efba85a7e14341de92f89b81cb1fc88d P b5de051fb2eb4725ba02a4ce070d1514 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e09a6f57e1b64b19b2ade354e28fa698 in term 1.
I20260926 04:02:18.749130 20410 leader_election.cc:302] T efba85a7e14341de92f89b81cb1fc88d P e09a6f57e1b64b19b2ade354e28fa698 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cee54717250642078a55edf53008319f, e09a6f57e1b64b19b2ade354e28fa698; no voters:
I20260926 04:02:18.749217 20543 raft_consensus.cc:2781] T efba85a7e14341de92f89b81cb1fc88d P e09a6f57e1b64b19b2ade354e28fa698 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:02:18.749359 20543 raft_consensus.cc:686] T efba85a7e14341de92f89b81cb1fc88d P e09a6f57e1b64b19b2ade354e28fa698 [term 1 LEADER]: Becoming Leader. State: Replica: e09a6f57e1b64b19b2ade354e28fa698, State: Running, Role: LEADER
I20260926 04:02:18.749466 20543 consensus_queue.cc:237] T efba85a7e14341de92f89b81cb1fc88d P e09a6f57e1b64b19b2ade354e28fa698 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee54717250642078a55edf53008319f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45941 } } peers { permanent_uuid: "e09a6f57e1b64b19b2ade354e28fa698" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43761 } } peers { permanent_uuid: "b5de051fb2eb4725ba02a4ce070d1514" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36505 } }
I20260926 04:02:18.750404 20070 catalog_manager.cc:5495] T efba85a7e14341de92f89b81cb1fc88d P e09a6f57e1b64b19b2ade354e28fa698 reported cstate change: term changed from 0 to 1, leader changed from <none> to e09a6f57e1b64b19b2ade354e28fa698 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "e09a6f57e1b64b19b2ade354e28fa698" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee54717250642078a55edf53008319f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e09a6f57e1b64b19b2ade354e28fa698" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b5de051fb2eb4725ba02a4ce070d1514" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36505 } health_report { overall_health: UNKNOWN } } }
W20260926 04:02:18.752363 20391 tablet.cc:2369] T efba85a7e14341de92f89b81cb1fc88d P b5de051fb2eb4725ba02a4ce070d1514: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 04:02:18.778354 20534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:18.775989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43802 (local address 127.0.98.3:43761)
0926 04:02:18.776052 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:18.776054 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:18.776067 (+ 13us) server_negotiation.cc:407] Connection header received
0926 04:02:18.776104 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:18.776105 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:18.776124 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:18.776187 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 04:02:18.776317 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:18.776810 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:18.777576 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:18.777753 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:18.777804 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:18.777994 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 04:02:18.778098 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 04:02:18.778161 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:18.778215 (+ 54us) server_negotiation.cc:299] Negotiation successful
0926 04:02:18.778232 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 04:02:18.780215 20213 raft_consensus.cc:1261] T efba85a7e14341de92f89b81cb1fc88d P cee54717250642078a55edf53008319f [term 1 FOLLOWER]: Refusing update from remote peer e09a6f57e1b64b19b2ade354e28fa698: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:02:18.780227 20344 raft_consensus.cc:1261] T efba85a7e14341de92f89b81cb1fc88d P b5de051fb2eb4725ba02a4ce070d1514 [term 1 FOLLOWER]: Refusing update from remote peer e09a6f57e1b64b19b2ade354e28fa698: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:02:18.780566 20545 consensus_queue.cc:1035] T efba85a7e14341de92f89b81cb1fc88d P e09a6f57e1b64b19b2ade354e28fa698 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5de051fb2eb4725ba02a4ce070d1514" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36505 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:02:18.780745 20543 consensus_queue.cc:1035] T efba85a7e14341de92f89b81cb1fc88d P e09a6f57e1b64b19b2ade354e28fa698 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cee54717250642078a55edf53008319f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:02:18.783095 20548 mvcc.cc:205] Tried to move back new op lower bound from 7333459307641196544 to 7333459307518648320. Current Snapshot: MvccSnapshot[applied={T|T < 7333459307641196544}]
I20260926 04:02:18.783360 20550 mvcc.cc:205] Tried to move back new op lower bound from 7333459307641196544 to 7333459307518648320. Current Snapshot: MvccSnapshot[applied={T|T < 7333459307641196544}]
I20260926 04:02:18.785840 20069 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 04:02:18.785943 20069 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 04:02:18.786705 20069 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5e5b1245d0e84a82904aeff063d23e03: Sending DeleteTablet for 3 replicas of tablet efba85a7e14341de92f89b81cb1fc88d
I20260926 04:02:18.787022 20193 tablet_service.cc:1514] Processing DeleteTablet for tablet efba85a7e14341de92f89b81cb1fc88d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:02:18 UTC) from {username='slave'} at 127.0.0.1:32810
I20260926 04:02:18.787117 20324 tablet_service.cc:1514] Processing DeleteTablet for tablet efba85a7e14341de92f89b81cb1fc88d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:02:18 UTC) from {username='slave'} at 127.0.0.1:59916
I20260926 04:02:18.787281 20448 tablet_service.cc:1514] Processing DeleteTablet for tablet efba85a7e14341de92f89b81cb1fc88d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 04:02:18 UTC) from {username='slave'} at 127.0.0.1:43786
I20260926 04:02:18.787315 20559 tablet_replica.cc:330] T efba85a7e14341de92f89b81cb1fc88d P b5de051fb2eb4725ba02a4ce070d1514: stopping tablet replica
I20260926 04:02:18.787315 20558 tablet_replica.cc:330] T efba85a7e14341de92f89b81cb1fc88d P cee54717250642078a55edf53008319f: stopping tablet replica
I20260926 04:02:18.787408 20558 raft_consensus.cc:2229] T efba85a7e14341de92f89b81cb1fc88d P cee54717250642078a55edf53008319f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 04:02:18.787418 20559 raft_consensus.cc:2229] T efba85a7e14341de92f89b81cb1fc88d P b5de051fb2eb4725ba02a4ce070d1514 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 04:02:18.787519 20559 raft_consensus.cc:2258] T efba85a7e14341de92f89b81cb1fc88d P b5de051fb2eb4725ba02a4ce070d1514 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:02:18.787518 20558 raft_consensus.cc:2258] T efba85a7e14341de92f89b81cb1fc88d P cee54717250642078a55edf53008319f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:02:18.787825 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 20131
I20260926 04:02:18.787757 20560 tablet_replica.cc:330] T efba85a7e14341de92f89b81cb1fc88d P e09a6f57e1b64b19b2ade354e28fa698: stopping tablet replica
I20260926 04:02:18.788738 20560 raft_consensus.cc:2229] T efba85a7e14341de92f89b81cb1fc88d P e09a6f57e1b64b19b2ade354e28fa698 [term 1 LEADER]: Raft consensus shutting down.
I20260926 04:02:18.788861 20560 raft_consensus.cc:2258] T efba85a7e14341de92f89b81cb1fc88d P e09a6f57e1b64b19b2ade354e28fa698 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 04:02:18.788993 20559 ts_tablet_manager.cc:1905] T efba85a7e14341de92f89b81cb1fc88d P b5de051fb2eb4725ba02a4ce070d1514: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:02:18.789279 20560 ts_tablet_manager.cc:1905] T efba85a7e14341de92f89b81cb1fc88d P e09a6f57e1b64b19b2ade354e28fa698: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 04:02:18.790781 20559 ts_tablet_manager.cc:1918] T efba85a7e14341de92f89b81cb1fc88d P b5de051fb2eb4725ba02a4ce070d1514: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 04:02:18.790860 20559 log.cc:1196] T efba85a7e14341de92f89b81cb1fc88d P b5de051fb2eb4725ba02a4ce070d1514: Deleting WAL directory at /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395095474643-392-0/minicluster-data/ts-1/wal/wals/efba85a7e14341de92f89b81cb1fc88d
I20260926 04:02:18.790882 20560 ts_tablet_manager.cc:1918] T efba85a7e14341de92f89b81cb1fc88d P e09a6f57e1b64b19b2ade354e28fa698: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 04:02:18.790948 20560 log.cc:1196] T efba85a7e14341de92f89b81cb1fc88d P e09a6f57e1b64b19b2ade354e28fa698: Deleting WAL directory at /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790395095474643-392-0/minicluster-data/ts-2/wal/wals/efba85a7e14341de92f89b81cb1fc88d
I20260926 04:02:18.791131 20559 ts_tablet_manager.cc:1939] T efba85a7e14341de92f89b81cb1fc88d P b5de051fb2eb4725ba02a4ce070d1514: Deleting consensus metadata
I20260926 04:02:18.791182 20560 ts_tablet_manager.cc:1939] T efba85a7e14341de92f89b81cb1fc88d P e09a6f57e1b64b19b2ade354e28fa698: Deleting consensus metadata
I20260926 04:02:18.791671 20058 catalog_manager.cc:4841] TS b5de051fb2eb4725ba02a4ce070d1514 (127.0.98.2:36505): tablet efba85a7e14341de92f89b81cb1fc88d (table test-table [id=d659155afae4463fae0c68ff5a0abc44]) successfully deleted
I20260926 04:02:18.792899 20057 catalog_manager.cc:4841] TS e09a6f57e1b64b19b2ade354e28fa698 (127.0.98.3:43761): tablet efba85a7e14341de92f89b81cb1fc88d (table test-table [id=d659155afae4463fae0c68ff5a0abc44]) successfully deleted
W20260926 04:02:18.793965 20057 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45941 (error 108)
I20260926 04:02:18.794171 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 20262
I20260926 04:02:18.794545 20533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:18.794415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45941 (local address 127.0.0.1:32820)
0926 04:02:18.794473 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:18.794507 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45941: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260926 04:02:18.794826 20057 catalog_manager.cc:4584] TS cee54717250642078a55edf53008319f (127.0.98.1:45941): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet efba85a7e14341de92f89b81cb1fc88d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45941: connect: Connection refused (error 111)
I20260926 04:02:18.799825 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 20393
I20260926 04:02:18.804917 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 20040
2026-09-26T04:02:18Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (758 ms)
[----------] 6 tests from EncryptionPolicyTest (8501 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790395095474643-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 04:02:18 dist-test-slave-nhpw krb5kdc[20564](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 04:02:18 dist-test-slave-nhpw krb5kdc[20564](info): set up 2 sockets
Sep 26 04:02:18 dist-test-slave-nhpw krb5kdc[20564](info): commencing operation
krb5kdc: starting...
W20260926 04:02:19.836531 392 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790395095474643-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790395095474643-392-0/krb5kdc/test-user.keytab.
Sep 26 04:02:19 dist-test-slave-nhpw krb5kdc[20564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T04:02:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:02:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790395095474643-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790395095474643-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 04:02:19.951987 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790395095474643-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790395095474643-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790395095474643-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35105
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790395095474643-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35105
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:02:20.031441 20577 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:02:20.031677 20577 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:02:20.031713 20577 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:02:20.032909 20577 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:02:20.032969 20577 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:02:20.033002 20577 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:02:20.033044 20577 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:02:20.033066 20577 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:02:20.034554 20577 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790395095474643-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790395095474643-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35105
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790395095474643-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35105
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20577
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790395095474643-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:02:20.034822 20577 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:02:20.035045 20577 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 04:02:20.039974 20577 server_base.cc:1034] running on GCE node
W20260926 04:02:20.039925 20585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:02:20.039937 20583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:02:20.039928 20582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:02:20.040380 20577 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:02:20.040627 20577 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:02:20.041836 20577 hybrid_clock.cc:648] HybridClock initialized: now 1790395340041830 us; error 43 us; skew 500 ppm
Sep 26 04:02:20 dist-test-slave-nhpw krb5kdc[20564](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395340, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:02:20.044226 20577 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 04:02:20.044755 20577 webserver.cc:460] Webserver started at http://127.0.0.1:41069/ using document root <none> and password file <none>
I20260926 04:02:20.044999 20577 fs_manager.cc:359] Metadata directory not provided
I20260926 04:02:20.045068 20577 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:02:20.045207 20577 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:02:20.046247 20577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790395095474643-392-0/minicluster-data/master-0/data/instance:
uuid: "f9ed8e08ec76482dbd066d4fc72a8494"
format_stamp: "Formatted at 2026-09-26 04:02:20 on dist-test-slave-nhpw"
server_key: "8da558619ef98bf774d3bf057d256a79"
server_key_iv: "88988773cf590c6cbf6c87c8d331d978"
server_key_version: "encryptionkey@0"
I20260926 04:02:20.046612 20577 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790395095474643-392-0/minicluster-data/master-0/wal/instance:
uuid: "f9ed8e08ec76482dbd066d4fc72a8494"
format_stamp: "Formatted at 2026-09-26 04:02:20 on dist-test-slave-nhpw"
server_key: "8da558619ef98bf774d3bf057d256a79"
server_key_iv: "88988773cf590c6cbf6c87c8d331d978"
server_key_version: "encryptionkey@0"
I20260926 04:02:20.047881 20577 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 04:02:20.048623 20592 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:20.049044 20577 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 04:02:20.049140 20577 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790395095474643-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790395095474643-392-0/minicluster-data/master-0/wal
uuid: "f9ed8e08ec76482dbd066d4fc72a8494"
format_stamp: "Formatted at 2026-09-26 04:02:20 on dist-test-slave-nhpw"
server_key: "8da558619ef98bf774d3bf057d256a79"
server_key_iv: "88988773cf590c6cbf6c87c8d331d978"
server_key_version: "encryptionkey@0"
I20260926 04:02:20.049220 20577 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790395095474643-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790395095474643-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790395095474643-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:02:20.073654 20577 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:02:20.075590 20577 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:02:20.075754 20577 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:02:20.079770 20577 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35105
I20260926 04:02:20.079805 20644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35105 every 8 connection(s)
I20260926 04:02:20.080176 20577 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
I20260926 04:02:20.081028 20645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:02:20.083190 20645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9ed8e08ec76482dbd066d4fc72a8494: Bootstrap starting.
I20260926 04:02:20.083751 20645 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9ed8e08ec76482dbd066d4fc72a8494: Neither blocks nor log segments found. Creating new log.
I20260926 04:02:20.084030 20645 log.cc:824] T 00000000000000000000000000000000 P f9ed8e08ec76482dbd066d4fc72a8494: Log is configured to *not* fsync() on all Append() calls
I20260926 04:02:20.084687 20645 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9ed8e08ec76482dbd066d4fc72a8494: No bootstrap required, opened a new log
I20260926 04:02:20.086180 20645 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9ed8e08ec76482dbd066d4fc72a8494 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9ed8e08ec76482dbd066d4fc72a8494" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35105 } }
I20260926 04:02:20.086306 20645 raft_consensus.cc:374] T 00000000000000000000000000000000 P f9ed8e08ec76482dbd066d4fc72a8494 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:02:20.086328 20645 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9ed8e08ec76482dbd066d4fc72a8494 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9ed8e08ec76482dbd066d4fc72a8494, State: Initialized, Role: FOLLOWER
I20260926 04:02:20.086530 20645 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9ed8e08ec76482dbd066d4fc72a8494 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9ed8e08ec76482dbd066d4fc72a8494" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35105 } }
I20260926 04:02:20.086615 20645 raft_consensus.cc:388] T 00000000000000000000000000000000 P f9ed8e08ec76482dbd066d4fc72a8494 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:02:20.086582 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 20577
I20260926 04:02:20.086639 20645 raft_consensus.cc:482] T 00000000000000000000000000000000 P f9ed8e08ec76482dbd066d4fc72a8494 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:02:20.086707 20645 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9ed8e08ec76482dbd066d4fc72a8494 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:02:20.086731 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790395095474643-392-0/minicluster-data/master-0/wal/instance
I20260926 04:02:20.086890 392 external_mini_cluster.cc:1428] Setting key a78f724bb4d3a1dd5ef9952f570f4053
I20260926 04:02:20.087301 20645 raft_consensus.cc:504] T 00000000000000000000000000000000 P f9ed8e08ec76482dbd066d4fc72a8494 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9ed8e08ec76482dbd066d4fc72a8494" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35105 } }
I20260926 04:02:20.087405 20645 leader_election.cc:302] T 00000000000000000000000000000000 P f9ed8e08ec76482dbd066d4fc72a8494 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f9ed8e08ec76482dbd066d4fc72a8494; no voters:
I20260926 04:02:20.087564 20645 leader_election.cc:288] T 00000000000000000000000000000000 P f9ed8e08ec76482dbd066d4fc72a8494 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 04:02:20.087595 20650 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f9ed8e08ec76482dbd066d4fc72a8494 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:02:20.087731 20650 raft_consensus.cc:686] T 00000000000000000000000000000000 P f9ed8e08ec76482dbd066d4fc72a8494 [term 1 LEADER]: Becoming Leader. State: Replica: f9ed8e08ec76482dbd066d4fc72a8494, State: Running, Role: LEADER
I20260926 04:02:20.087810 20645 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9ed8e08ec76482dbd066d4fc72a8494 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:02:20.087808 20650 consensus_queue.cc:237] T 00000000000000000000000000000000 P f9ed8e08ec76482dbd066d4fc72a8494 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9ed8e08ec76482dbd066d4fc72a8494" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35105 } }
I20260926 04:02:20.088587 20650 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9ed8e08ec76482dbd066d4fc72a8494 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9ed8e08ec76482dbd066d4fc72a8494. Latest consensus state: current_term: 1 leader_uuid: "f9ed8e08ec76482dbd066d4fc72a8494" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9ed8e08ec76482dbd066d4fc72a8494" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35105 } } }
I20260926 04:02:20.088687 20650 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9ed8e08ec76482dbd066d4fc72a8494 [sys.catalog]: This master's current role is: LEADER
I20260926 04:02:20.088995 20652 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9ed8e08ec76482dbd066d4fc72a8494 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f9ed8e08ec76482dbd066d4fc72a8494" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9ed8e08ec76482dbd066d4fc72a8494" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35105 } } }
I20260926 04:02:20.089077 20652 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9ed8e08ec76482dbd066d4fc72a8494 [sys.catalog]: This master's current role is: LEADER
I20260926 04:02:20.089200 20659 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:02:20.089742 20659 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:02:20.091236 20659 catalog_manager.cc:1347] Generated new cluster ID: 332f5b47dca449beae1a4782f23d2e2d
I20260926 04:02:20.091291 20659 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 04:02:20 dist-test-slave-nhpw krb5kdc[20564](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 04:02:20.096684 20659 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:02:20.097373 20659 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:02:20.098162 20649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:20.087376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43565 (local address 127.0.0.1:35105)
0926 04:02:20.087566 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:20.087573 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:20.087599 (+ 26us) server_negotiation.cc:407] Connection header received
0926 04:02:20.087756 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:20.087763 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:20.087821 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:20.088008 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:20.088875 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:20.090202 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:20.091055 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:20.091629 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:20.091724 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:20.094480 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:20.094492 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:20.094492 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:02:20.094503 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:20.096467 (+ 1964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:20.096973 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:20.096976 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:20.096976 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:20.097063 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:20.097308 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:20.097309 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:20.097310 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:20.097568 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:20.097764 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:20.097857 (+ 93us) server_negotiation.cc:299] Negotiation successful
0926 04:02:20.097982 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":87,"threads_started":1}
I20260926 04:02:20.099004 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 04:02:20.106163 20659 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f9ed8e08ec76482dbd066d4fc72a8494: Generated new TSK 0
I20260926 04:02:20.106323 20659 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 04:02:20.111069 20649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:20.100926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46359 (local address 127.0.0.1:35105)
0926 04:02:20.100965 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:20.100967 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:20.104810 (+ 3843us) server_negotiation.cc:407] Connection header received
0926 04:02:20.104881 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:20.104882 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:20.104892 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:20.104936 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:20.105509 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:20.106215 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:20.106967 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:20.107318 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:20.107397 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:20.108508 (+ 1111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:20.108509 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:20.108510 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:02:20.108516 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:20.110015 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:20.110350 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:20.110351 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:20.110352 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:20.110407 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:20.110601 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:20.110602 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:20.110602 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:20.110726 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:20.110825 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:20.110917 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 04:02:20.110976 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 04:02:20.113013 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 20577
W20260926 04:02:20.117262 20562 connection.cc:535] client connection to 127.0.0.1:35105 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35105): Connection reset by peer (error 104)
2026-09-26T04:02:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1305 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790395095474643-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 04:02:20 dist-test-slave-nhpw krb5kdc[20676](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 04:02:20 dist-test-slave-nhpw krb5kdc[20676](info): set up 2 sockets
Sep 26 04:02:20 dist-test-slave-nhpw krb5kdc[20676](info): commencing operation
krb5kdc: starting...
W20260926 04:02:21.140574 392 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790395095474643-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790395095474643-392-0/krb5kdc/test-user.keytab.
Sep 26 04:02:21 dist-test-slave-nhpw krb5kdc[20676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395341, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T04:02:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:02:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790395095474643-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790395095474643-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 04:02:21.260268 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790395095474643-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790395095474643-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790395095474643-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35721
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790395095474643-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35721
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:02:21.342399 20689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:02:21.342653 20689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:02:21.342694 20689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:02:21.343865 20689 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:02:21.343941 20689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:02:21.343966 20689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:02:21.344005 20689 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:02:21.344038 20689 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:02:21.345556 20689 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790395095474643-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790395095474643-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35721
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790395095474643-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20689
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790395095474643-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:02:21.345830 20689 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:02:21.346058 20689 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:02:21.350915 20694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:02:21.350946 20695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:02:21.350977 20697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:02:21.351017 20689 server_base.cc:1034] running on GCE node
I20260926 04:02:21.351383 20689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:02:21.351632 20689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:02:21.352775 20689 hybrid_clock.cc:648] HybridClock initialized: now 1790395341352748 us; error 43 us; skew 500 ppm
Sep 26 04:02:21 dist-test-slave-nhpw krb5kdc[20676](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395341, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:02:21.359563 20689 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 04:02:21.360102 20689 webserver.cc:460] Webserver started at http://127.0.0.1:39139/ using document root <none> and password file <none>
I20260926 04:02:21.360338 20689 fs_manager.cc:359] Metadata directory not provided
I20260926 04:02:21.360406 20689 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:02:21.360569 20689 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:02:21.361793 20689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790395095474643-392-0/minicluster-data/master-0/data/instance:
uuid: "2348a6bef87b48d581351e275263e34b"
format_stamp: "Formatted at 2026-09-26 04:02:21 on dist-test-slave-nhpw"
server_key: "fd48ef6bb0db1da4747415b62ae157c1"
server_key_iv: "cb69d0ecfb83ab8b4d977c48098cc7fd"
server_key_version: "encryptionkey@0"
I20260926 04:02:21.362166 20689 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790395095474643-392-0/minicluster-data/master-0/wal/instance:
uuid: "2348a6bef87b48d581351e275263e34b"
format_stamp: "Formatted at 2026-09-26 04:02:21 on dist-test-slave-nhpw"
server_key: "fd48ef6bb0db1da4747415b62ae157c1"
server_key_iv: "cb69d0ecfb83ab8b4d977c48098cc7fd"
server_key_version: "encryptionkey@0"
I20260926 04:02:21.363593 20689 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 04:02:21.364363 20704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:21.364593 20689 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:02:21.364728 20689 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790395095474643-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790395095474643-392-0/minicluster-data/master-0/wal
uuid: "2348a6bef87b48d581351e275263e34b"
format_stamp: "Formatted at 2026-09-26 04:02:21 on dist-test-slave-nhpw"
server_key: "fd48ef6bb0db1da4747415b62ae157c1"
server_key_iv: "cb69d0ecfb83ab8b4d977c48098cc7fd"
server_key_version: "encryptionkey@0"
I20260926 04:02:21.364817 20689 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790395095474643-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790395095474643-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790395095474643-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:02:21.378388 20689 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:02:21.380468 20689 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:02:21.380687 20689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:02:21.384928 20689 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35721
I20260926 04:02:21.385011 20756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35721 every 8 connection(s)
I20260926 04:02:21.385371 20689 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
I20260926 04:02:21.386197 20757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:02:21.388420 20757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2348a6bef87b48d581351e275263e34b: Bootstrap starting.
I20260926 04:02:21.389134 20757 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2348a6bef87b48d581351e275263e34b: Neither blocks nor log segments found. Creating new log.
I20260926 04:02:21.389375 20757 log.cc:824] T 00000000000000000000000000000000 P 2348a6bef87b48d581351e275263e34b: Log is configured to *not* fsync() on all Append() calls
I20260926 04:02:21.390053 20757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2348a6bef87b48d581351e275263e34b: No bootstrap required, opened a new log
I20260926 04:02:21.391577 20757 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2348a6bef87b48d581351e275263e34b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2348a6bef87b48d581351e275263e34b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35721 } }
I20260926 04:02:21.391716 20757 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2348a6bef87b48d581351e275263e34b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:02:21.391739 20757 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2348a6bef87b48d581351e275263e34b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2348a6bef87b48d581351e275263e34b, State: Initialized, Role: FOLLOWER
I20260926 04:02:21.391887 20757 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2348a6bef87b48d581351e275263e34b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2348a6bef87b48d581351e275263e34b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35721 } }
I20260926 04:02:21.391968 20757 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2348a6bef87b48d581351e275263e34b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:02:21.391996 20757 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2348a6bef87b48d581351e275263e34b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:02:21.392025 20757 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2348a6bef87b48d581351e275263e34b [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:02:21.392517 20757 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2348a6bef87b48d581351e275263e34b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2348a6bef87b48d581351e275263e34b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35721 } }
I20260926 04:02:21.392609 20757 leader_election.cc:302] T 00000000000000000000000000000000 P 2348a6bef87b48d581351e275263e34b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2348a6bef87b48d581351e275263e34b; no voters:
I20260926 04:02:21.392802 20757 leader_election.cc:288] T 00000000000000000000000000000000 P 2348a6bef87b48d581351e275263e34b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 04:02:21.392879 20760 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2348a6bef87b48d581351e275263e34b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:02:21.393074 20760 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2348a6bef87b48d581351e275263e34b [term 1 LEADER]: Becoming Leader. State: Replica: 2348a6bef87b48d581351e275263e34b, State: Running, Role: LEADER
I20260926 04:02:21.393082 20757 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2348a6bef87b48d581351e275263e34b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:02:21.393175 20760 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2348a6bef87b48d581351e275263e34b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2348a6bef87b48d581351e275263e34b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35721 } }
I20260926 04:02:21.393612 20762 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2348a6bef87b48d581351e275263e34b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2348a6bef87b48d581351e275263e34b. Latest consensus state: current_term: 1 leader_uuid: "2348a6bef87b48d581351e275263e34b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2348a6bef87b48d581351e275263e34b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35721 } } }
I20260926 04:02:21.393639 20761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2348a6bef87b48d581351e275263e34b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2348a6bef87b48d581351e275263e34b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2348a6bef87b48d581351e275263e34b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35721 } } }
I20260926 04:02:21.393790 20761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2348a6bef87b48d581351e275263e34b [sys.catalog]: This master's current role is: LEADER
I20260926 04:02:21.393756 20762 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2348a6bef87b48d581351e275263e34b [sys.catalog]: This master's current role is: LEADER
I20260926 04:02:21.394182 20764 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:02:21.394783 20764 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:02:21.395200 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 20689
I20260926 04:02:21.395339 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790395095474643-392-0/minicluster-data/master-0/wal/instance
I20260926 04:02:21.395565 392 external_mini_cluster.cc:1428] Setting key d762c5419af1378e5e5e3f9c00cb7deb
I20260926 04:02:21.397600 20764 catalog_manager.cc:1347] Generated new cluster ID: 49151601e3db498fafbc396adf550d7e
I20260926 04:02:21.397662 20764 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 04:02:21.401516 20779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:21.396029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55011 (local address 127.0.0.1:35721)
0926 04:02:21.396785 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:21.396793 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:21.396830 (+ 37us) server_negotiation.cc:407] Connection header received
0926 04:02:21.396944 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:21.396952 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:21.397015 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:21.397167 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:21.397445 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:21.398547 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:21.399415 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:21.399863 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:21.399998 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:21.400752 (+ 754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:21.400757 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:21.400757 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:02:21.400938 (+ 181us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:21.400969 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:02:21.400980 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:21.401088 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:21.401166 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 04:02:21.401196 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":708,"thread_start_us":90,"threads_started":1}
I20260926 04:02:21.402474 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 04:02:21.406893 20779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:21.403889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49217 (local address 127.0.0.1:35721)
0926 04:02:21.403925 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:21.403927 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:21.404058 (+ 131us) server_negotiation.cc:407] Connection header received
0926 04:02:21.404133 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:21.404134 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:21.404145 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:21.404191 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:21.404325 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:21.404929 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:21.405646 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:21.405859 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:21.405933 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:21.406645 (+ 712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:21.406647 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:21.406647 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:02:21.406652 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:21.406666 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:02:21.406670 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:21.406724 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:21.406782 (+ 58us) server_negotiation.cc:299] Negotiation successful
0926 04:02:21.406795 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 04:02:21.413488 20764 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:02:21.414026 20764 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:02:21.419625 20764 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2348a6bef87b48d581351e275263e34b: Generated new TSK 0
I20260926 04:02:21.419827 20764 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 04:02:21.423924 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 20689
W20260926 04:02:21.427861 20674 connection.cc:535] client connection to 127.0.0.1:35721 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35721): Connection reset by peer (error 104)
2026-09-26T04:02:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1310 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790395095474643-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 04:02:21 dist-test-slave-nhpw krb5kdc[20788](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 04:02:21 dist-test-slave-nhpw krb5kdc[20788](info): set up 2 sockets
Sep 26 04:02:21 dist-test-slave-nhpw krb5kdc[20788](info): commencing operation
krb5kdc: starting...
W20260926 04:02:22.456833 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790395095474643-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790395095474643-392-0/krb5kdc/test-user.keytab.
Sep 26 04:02:22 dist-test-slave-nhpw krb5kdc[20788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T04:02:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:02:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790395095474643-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790395095474643-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 04:02:22.578672 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790395095474643-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790395095474643-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790395095474643-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43259
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790395095474643-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43259
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:02:22.661480 20801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:02:22.661703 20801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:02:22.661741 20801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:02:22.662853 20801 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:02:22.662917 20801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:02:22.662941 20801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:02:22.662964 20801 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:02:22.662992 20801 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:02:22.664422 20801 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790395095474643-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790395095474643-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43259
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790395095474643-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43259
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20801
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790395095474643-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:02:22.664710 20801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:02:22.664928 20801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:02:22.669854 20806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:02:22.669872 20809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:02:22.669902 20807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:02:22.670231 20801 server_base.cc:1034] running on GCE node
I20260926 04:02:22.670480 20801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:02:22.670780 20801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:02:22.671963 20801 hybrid_clock.cc:648] HybridClock initialized: now 1790395342671934 us; error 58 us; skew 500 ppm
Sep 26 04:02:22 dist-test-slave-nhpw krb5kdc[20788](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395342, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:02:22.674644 20801 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 04:02:22.675173 20801 webserver.cc:460] Webserver started at http://127.0.0.1:43567/ using document root <none> and password file <none>
I20260926 04:02:22.675428 20801 fs_manager.cc:359] Metadata directory not provided
I20260926 04:02:22.675498 20801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:02:22.675642 20801 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:02:22.676761 20801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790395095474643-392-0/minicluster-data/master-0/data/instance:
uuid: "b49b7f551f0a45249add2e2f21b8fab6"
format_stamp: "Formatted at 2026-09-26 04:02:22 on dist-test-slave-nhpw"
server_key: "a9b4b7586d020602a4fb8d709afa8e38"
server_key_iv: "e09249c04e9a00179eaa4e56b1f9c76a"
server_key_version: "encryptionkey@0"
I20260926 04:02:22.677124 20801 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790395095474643-392-0/minicluster-data/master-0/wal/instance:
uuid: "b49b7f551f0a45249add2e2f21b8fab6"
format_stamp: "Formatted at 2026-09-26 04:02:22 on dist-test-slave-nhpw"
server_key: "a9b4b7586d020602a4fb8d709afa8e38"
server_key_iv: "e09249c04e9a00179eaa4e56b1f9c76a"
server_key_version: "encryptionkey@0"
I20260926 04:02:22.678437 20801 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 04:02:22.679234 20816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:22.679407 20801 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:02:22.679507 20801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790395095474643-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790395095474643-392-0/minicluster-data/master-0/wal
uuid: "b49b7f551f0a45249add2e2f21b8fab6"
format_stamp: "Formatted at 2026-09-26 04:02:22 on dist-test-slave-nhpw"
server_key: "a9b4b7586d020602a4fb8d709afa8e38"
server_key_iv: "e09249c04e9a00179eaa4e56b1f9c76a"
server_key_version: "encryptionkey@0"
I20260926 04:02:22.679605 20801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790395095474643-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790395095474643-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790395095474643-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:02:22.682953 20801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:02:22.685247 20801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:02:22.685405 20801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:02:22.689357 20801 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43259
I20260926 04:02:22.689388 20868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43259 every 8 connection(s)
I20260926 04:02:22.689718 20801 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
I20260926 04:02:22.690675 20869 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:02:22.693034 20869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b49b7f551f0a45249add2e2f21b8fab6: Bootstrap starting.
I20260926 04:02:22.693447 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 20801
I20260926 04:02:22.693578 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790395095474643-392-0/minicluster-data/master-0/wal/instance
I20260926 04:02:22.693782 392 external_mini_cluster.cc:1428] Setting key 839e9d7247282c288ed1a75ab0d0a412
I20260926 04:02:22.693783 20869 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b49b7f551f0a45249add2e2f21b8fab6: Neither blocks nor log segments found. Creating new log.
I20260926 04:02:22.694126 20869 log.cc:824] T 00000000000000000000000000000000 P b49b7f551f0a45249add2e2f21b8fab6: Log is configured to *not* fsync() on all Append() calls
I20260926 04:02:22.694895 20869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b49b7f551f0a45249add2e2f21b8fab6: No bootstrap required, opened a new log
I20260926 04:02:22.696293 20873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:22.694298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39645 (local address 127.0.0.1:43259)
0926 04:02:22.694474 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:22.694483 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:22.694519 (+ 36us) server_negotiation.cc:407] Connection header received
0926 04:02:22.695056 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:22.695063 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:22.695091 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:22.695213 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:22.695859 (+ 646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:22.695862 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:22.695862 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:02:22.695866 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:22.695902 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:02:22.695913 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:22.695979 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:22.696082 (+ 103us) server_negotiation.cc:299] Negotiation successful
0926 04:02:22.696127 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":76,"threads_started":1}
I20260926 04:02:22.697638 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 04:02:22.697566 20869 raft_consensus.cc:348] T 00000000000000000000000000000000 P b49b7f551f0a45249add2e2f21b8fab6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b49b7f551f0a45249add2e2f21b8fab6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43259 } }
I20260926 04:02:22.697713 20869 raft_consensus.cc:374] T 00000000000000000000000000000000 P b49b7f551f0a45249add2e2f21b8fab6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:02:22.697760 20869 raft_consensus.cc:729] T 00000000000000000000000000000000 P b49b7f551f0a45249add2e2f21b8fab6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b49b7f551f0a45249add2e2f21b8fab6, State: Initialized, Role: FOLLOWER
I20260926 04:02:22.697881 20869 consensus_queue.cc:260] T 00000000000000000000000000000000 P b49b7f551f0a45249add2e2f21b8fab6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b49b7f551f0a45249add2e2f21b8fab6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43259 } }
I20260926 04:02:22.697999 20869 raft_consensus.cc:388] T 00000000000000000000000000000000 P b49b7f551f0a45249add2e2f21b8fab6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:02:22.698045 20869 raft_consensus.cc:482] T 00000000000000000000000000000000 P b49b7f551f0a45249add2e2f21b8fab6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:02:22.698101 20869 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b49b7f551f0a45249add2e2f21b8fab6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:02:22.698657 20869 raft_consensus.cc:504] T 00000000000000000000000000000000 P b49b7f551f0a45249add2e2f21b8fab6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b49b7f551f0a45249add2e2f21b8fab6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43259 } }
I20260926 04:02:22.698776 20869 leader_election.cc:302] T 00000000000000000000000000000000 P b49b7f551f0a45249add2e2f21b8fab6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b49b7f551f0a45249add2e2f21b8fab6; no voters:
I20260926 04:02:22.698959 20869 leader_election.cc:288] T 00000000000000000000000000000000 P b49b7f551f0a45249add2e2f21b8fab6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 04:02:22.698994 20878 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b49b7f551f0a45249add2e2f21b8fab6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:02:22.699178 20878 raft_consensus.cc:686] T 00000000000000000000000000000000 P b49b7f551f0a45249add2e2f21b8fab6 [term 1 LEADER]: Becoming Leader. State: Replica: b49b7f551f0a45249add2e2f21b8fab6, State: Running, Role: LEADER
I20260926 04:02:22.699211 20869 sys_catalog.cc:564] T 00000000000000000000000000000000 P b49b7f551f0a45249add2e2f21b8fab6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:02:22.699383 20878 consensus_queue.cc:237] T 00000000000000000000000000000000 P b49b7f551f0a45249add2e2f21b8fab6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b49b7f551f0a45249add2e2f21b8fab6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43259 } }
I20260926 04:02:22.699769 20878 sys_catalog.cc:455] T 00000000000000000000000000000000 P b49b7f551f0a45249add2e2f21b8fab6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b49b7f551f0a45249add2e2f21b8fab6. Latest consensus state: current_term: 1 leader_uuid: "b49b7f551f0a45249add2e2f21b8fab6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b49b7f551f0a45249add2e2f21b8fab6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43259 } } }
I20260926 04:02:22.699807 20879 sys_catalog.cc:455] T 00000000000000000000000000000000 P b49b7f551f0a45249add2e2f21b8fab6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b49b7f551f0a45249add2e2f21b8fab6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b49b7f551f0a45249add2e2f21b8fab6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43259 } } }
I20260926 04:02:22.699893 20878 sys_catalog.cc:458] T 00000000000000000000000000000000 P b49b7f551f0a45249add2e2f21b8fab6 [sys.catalog]: This master's current role is: LEADER
I20260926 04:02:22.700212 20888 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:02:22.700906 20888 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:02:22.700948 20879 sys_catalog.cc:458] T 00000000000000000000000000000000 P b49b7f551f0a45249add2e2f21b8fab6 [sys.catalog]: This master's current role is: LEADER
I20260926 04:02:22.701033 20873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:22.699263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41091 (local address 127.0.0.1:43259)
0926 04:02:22.699299 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:22.699301 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:22.699340 (+ 39us) server_negotiation.cc:407] Connection header received
0926 04:02:22.699401 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:22.699402 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:22.699406 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:22.699470 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:22.700342 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:22.700344 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:22.700344 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:02:22.700349 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:22.700365 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:02:22.700369 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:22.700403 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:22.700854 (+ 451us) server_negotiation.cc:299] Negotiation successful
0926 04:02:22.700869 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 04:02:22.702531 20888 catalog_manager.cc:1347] Generated new cluster ID: 3cf0474da9ec470891953be76e42b2cf
I20260926 04:02:22.702607 20888 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 04:02:22.728866 20888 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:02:22.729087 20888 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:02:22.742311 20888 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b49b7f551f0a45249add2e2f21b8fab6: Generated new TSK 0
I20260926 04:02:22.742525 20888 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 04:02:22.753953 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 20801
2026-09-26T04:02:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1330 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790395095474643-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 04:02:22 dist-test-slave-nhpw krb5kdc[20900](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 04:02:22 dist-test-slave-nhpw krb5kdc[20900](info): set up 2 sockets
Sep 26 04:02:22 dist-test-slave-nhpw krb5kdc[20900](info): commencing operation
krb5kdc: starting...
W20260926 04:02:23.828720 392 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790395095474643-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790395095474643-392-0/krb5kdc/test-user.keytab.
Sep 26 04:02:23 dist-test-slave-nhpw krb5kdc[20900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395343, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T04:02:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:02:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790395095474643-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790395095474643-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 04:02:23.947702 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790395095474643-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790395095474643-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790395095474643-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45461
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790395095474643-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45461
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:02:24.026517 20913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:02:24.026790 20913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:02:24.026831 20913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:02:24.027977 20913 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:02:24.028045 20913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:02:24.028069 20913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:02:24.028096 20913 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:02:24.028127 20913 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:02:24.029587 20913 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790395095474643-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790395095474643-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45461
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790395095474643-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20913
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790395095474643-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:02:24.029846 20913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:02:24.030077 20913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:02:24.034873 20921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:02:24.034893 20919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:02:24.034883 20918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:02:24.035140 20913 server_base.cc:1034] running on GCE node
I20260926 04:02:24.035358 20913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:02:24.035604 20913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:02:24.036796 20913 hybrid_clock.cc:648] HybridClock initialized: now 1790395344036786 us; error 38 us; skew 500 ppm
Sep 26 04:02:24 dist-test-slave-nhpw krb5kdc[20900](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395344, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:02:24.039222 20913 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 04:02:24.039585 20913 webserver.cc:460] Webserver started at http://127.0.0.1:38687/ using document root <none> and password file <none>
I20260926 04:02:24.039810 20913 fs_manager.cc:359] Metadata directory not provided
I20260926 04:02:24.039855 20913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:02:24.039940 20913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:02:24.040982 20913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790395095474643-392-0/minicluster-data/master-0/data/instance:
uuid: "0873dc3d6aaf4c59a7fc907a28da4e3e"
format_stamp: "Formatted at 2026-09-26 04:02:24 on dist-test-slave-nhpw"
server_key: "87d595a5a20678483f66286bfad83ba6"
server_key_iv: "9be10f196b8bcb2770a2d66261090678"
server_key_version: "encryptionkey@0"
I20260926 04:02:24.041281 20913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790395095474643-392-0/minicluster-data/master-0/wal/instance:
uuid: "0873dc3d6aaf4c59a7fc907a28da4e3e"
format_stamp: "Formatted at 2026-09-26 04:02:24 on dist-test-slave-nhpw"
server_key: "87d595a5a20678483f66286bfad83ba6"
server_key_iv: "9be10f196b8bcb2770a2d66261090678"
server_key_version: "encryptionkey@0"
I20260926 04:02:24.042554 20913 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 04:02:24.043365 20928 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:24.043588 20913 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:02:24.043707 20913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790395095474643-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790395095474643-392-0/minicluster-data/master-0/wal
uuid: "0873dc3d6aaf4c59a7fc907a28da4e3e"
format_stamp: "Formatted at 2026-09-26 04:02:24 on dist-test-slave-nhpw"
server_key: "87d595a5a20678483f66286bfad83ba6"
server_key_iv: "9be10f196b8bcb2770a2d66261090678"
server_key_version: "encryptionkey@0"
I20260926 04:02:24.043790 20913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790395095474643-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790395095474643-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790395095474643-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:02:24.063371 20913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:02:24.065925 20913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:02:24.066114 20913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:02:24.070472 20913 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45461
I20260926 04:02:24.070504 20980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45461 every 8 connection(s)
I20260926 04:02:24.070875 20913 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
I20260926 04:02:24.071767 20981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:02:24.072089 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 20913
I20260926 04:02:24.072237 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790395095474643-392-0/minicluster-data/master-0/wal/instance
I20260926 04:02:24.072429 392 external_mini_cluster.cc:1428] Setting key adffbf8f882c5262154c0241d0f2118c
I20260926 04:02:24.074837 20981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0873dc3d6aaf4c59a7fc907a28da4e3e: Bootstrap starting.
I20260926 04:02:24.075490 20981 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0873dc3d6aaf4c59a7fc907a28da4e3e: Neither blocks nor log segments found. Creating new log.
I20260926 04:02:24.075791 20981 log.cc:824] T 00000000000000000000000000000000 P 0873dc3d6aaf4c59a7fc907a28da4e3e: Log is configured to *not* fsync() on all Append() calls
I20260926 04:02:24.076480 20981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0873dc3d6aaf4c59a7fc907a28da4e3e: No bootstrap required, opened a new log
I20260926 04:02:24.078626 20981 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0873dc3d6aaf4c59a7fc907a28da4e3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0873dc3d6aaf4c59a7fc907a28da4e3e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45461 } }
I20260926 04:02:24.078752 20981 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0873dc3d6aaf4c59a7fc907a28da4e3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:02:24.078774 20981 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0873dc3d6aaf4c59a7fc907a28da4e3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0873dc3d6aaf4c59a7fc907a28da4e3e, State: Initialized, Role: FOLLOWER
I20260926 04:02:24.078907 20981 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0873dc3d6aaf4c59a7fc907a28da4e3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0873dc3d6aaf4c59a7fc907a28da4e3e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45461 } }
I20260926 04:02:24.078976 20981 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0873dc3d6aaf4c59a7fc907a28da4e3e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:02:24.079002 20981 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0873dc3d6aaf4c59a7fc907a28da4e3e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:02:24.079077 20981 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0873dc3d6aaf4c59a7fc907a28da4e3e [term 0 FOLLOWER]: Advancing to term 1
Sep 26 04:02:24 dist-test-slave-nhpw krb5kdc[20900](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395343, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 04:02:24.079563 20981 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0873dc3d6aaf4c59a7fc907a28da4e3e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0873dc3d6aaf4c59a7fc907a28da4e3e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45461 } }
I20260926 04:02:24.079635 20981 leader_election.cc:302] T 00000000000000000000000000000000 P 0873dc3d6aaf4c59a7fc907a28da4e3e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0873dc3d6aaf4c59a7fc907a28da4e3e; no voters:
I20260926 04:02:24.079797 20981 leader_election.cc:288] T 00000000000000000000000000000000 P 0873dc3d6aaf4c59a7fc907a28da4e3e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 04:02:24.079850 20986 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0873dc3d6aaf4c59a7fc907a28da4e3e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:02:24.079998 20986 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0873dc3d6aaf4c59a7fc907a28da4e3e [term 1 LEADER]: Becoming Leader. State: Replica: 0873dc3d6aaf4c59a7fc907a28da4e3e, State: Running, Role: LEADER
I20260926 04:02:24.080052 20981 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0873dc3d6aaf4c59a7fc907a28da4e3e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:02:24.080168 20986 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0873dc3d6aaf4c59a7fc907a28da4e3e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0873dc3d6aaf4c59a7fc907a28da4e3e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45461 } }
I20260926 04:02:24.080576 20988 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0873dc3d6aaf4c59a7fc907a28da4e3e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0873dc3d6aaf4c59a7fc907a28da4e3e. Latest consensus state: current_term: 1 leader_uuid: "0873dc3d6aaf4c59a7fc907a28da4e3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0873dc3d6aaf4c59a7fc907a28da4e3e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45461 } } }
I20260926 04:02:24.080586 20987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0873dc3d6aaf4c59a7fc907a28da4e3e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0873dc3d6aaf4c59a7fc907a28da4e3e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0873dc3d6aaf4c59a7fc907a28da4e3e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45461 } } }
I20260926 04:02:24.080685 20988 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0873dc3d6aaf4c59a7fc907a28da4e3e [sys.catalog]: This master's current role is: LEADER
I20260926 04:02:24.080686 20987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0873dc3d6aaf4c59a7fc907a28da4e3e [sys.catalog]: This master's current role is: LEADER
W20260926 04:02:24.082082 21002 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0873dc3d6aaf4c59a7fc907a28da4e3e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 04:02:24.082139 21002 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 04:02:24.082201 20994 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:02:24.082355 20994 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:02:24.084020 20984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:24.072984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38039 (local address 127.0.0.1:45461)
0926 04:02:24.073190 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:24.073199 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:24.073233 (+ 34us) server_negotiation.cc:407] Connection header received
0926 04:02:24.073761 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:24.073769 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:24.073828 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:24.073975 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:24.074635 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:24.075990 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:24.076943 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:24.077584 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:24.077692 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:24.080178 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:24.080182 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:24.080183 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:02:24.080195 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:24.082350 (+ 2155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:24.082788 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:24.082791 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:24.082792 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:24.082899 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:24.083122 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:24.083123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:24.083123 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:24.083442 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:24.083624 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:24.083689 (+ 65us) server_negotiation.cc:299] Negotiation successful
0926 04:02:24.083824 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":86,"threads_started":1}
I20260926 04:02:24.084172 20994 catalog_manager.cc:1347] Generated new cluster ID: 89ccf605f1314551b1c097dccac1a302
I20260926 04:02:24.084205 20994 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 04:02:24.084794 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 04:02:24.092831 20984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:24.086143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42451 (local address 127.0.0.1:45461)
0926 04:02:24.086174 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:24.086175 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:24.086279 (+ 104us) server_negotiation.cc:407] Connection header received
0926 04:02:24.086356 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:24.086357 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:24.086365 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:24.086389 (+ 24us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:24.086981 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:24.087685 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:24.088622 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:24.088950 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:24.088985 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:24.090305 (+ 1320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:24.090306 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:24.090307 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:02:24.090313 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:24.091756 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:24.092090 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:24.092091 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:24.092091 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:24.092135 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:24.092292 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:24.092292 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:24.092293 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:24.092394 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:24.092455 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:24.092672 (+ 217us) server_negotiation.cc:299] Negotiation successful
0926 04:02:24.092740 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260926 04:02:24.095289 20994 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:02:24.095824 20994 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:02:24.100064 20994 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0873dc3d6aaf4c59a7fc907a28da4e3e: Generated new TSK 0
I20260926 04:02:24.100274 20994 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 04:02:24.103644 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 20913
2026-09-26T04:02:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1349 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790395095474643-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 04:02:24 dist-test-slave-nhpw krb5kdc[21012](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 04:02:24 dist-test-slave-nhpw krb5kdc[21012](info): set up 2 sockets
Sep 26 04:02:24 dist-test-slave-nhpw krb5kdc[21012](info): commencing operation
krb5kdc: starting...
W20260926 04:02:25.136719 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790395095474643-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790395095474643-392-0/krb5kdc/test-user.keytab.
Sep 26 04:02:25 dist-test-slave-nhpw krb5kdc[21012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395345, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T04:02:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:02:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790395095474643-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790395095474643-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 04:02:25.252254 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790395095474643-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790395095474643-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790395095474643-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34009
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790395095474643-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34009
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:02:25.331502 21025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:02:25.331689 21025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:02:25.331712 21025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:02:25.332849 21025 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:02:25.332888 21025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:02:25.332902 21025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:02:25.332914 21025 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:02:25.332926 21025 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:02:25.334343 21025 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790395095474643-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790395095474643-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34009
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790395095474643-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34009
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790395095474643-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:02:25.334578 21025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:02:25.334802 21025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:02:25.339527 21030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:02:25.339515 21033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:02:25.339701 21025 server_base.cc:1034] running on GCE node
W20260926 04:02:25.339515 21031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:02:25.340046 21025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:02:25.340376 21025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:02:25.341550 21025 hybrid_clock.cc:648] HybridClock initialized: now 1790395345341531 us; error 39 us; skew 500 ppm
Sep 26 04:02:25 dist-test-slave-nhpw krb5kdc[21012](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395345, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:02:25.343986 21025 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 04:02:25.344416 21025 webserver.cc:460] Webserver started at http://127.0.0.1:38245/ using document root <none> and password file <none>
I20260926 04:02:25.344717 21025 fs_manager.cc:359] Metadata directory not provided
I20260926 04:02:25.344786 21025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:02:25.344944 21025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:02:25.345993 21025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790395095474643-392-0/minicluster-data/master-0/data/instance:
uuid: "1a5584d00a3d41bc9d0a874b861737a3"
format_stamp: "Formatted at 2026-09-26 04:02:25 on dist-test-slave-nhpw"
server_key: "0fbf543ff39d28fd64f36b111ebe8f05"
server_key_iv: "66f8e40556be05276825e5f83bd77311"
server_key_version: "encryptionkey@0"
I20260926 04:02:25.346370 21025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790395095474643-392-0/minicluster-data/master-0/wal/instance:
uuid: "1a5584d00a3d41bc9d0a874b861737a3"
format_stamp: "Formatted at 2026-09-26 04:02:25 on dist-test-slave-nhpw"
server_key: "0fbf543ff39d28fd64f36b111ebe8f05"
server_key_iv: "66f8e40556be05276825e5f83bd77311"
server_key_version: "encryptionkey@0"
I20260926 04:02:25.347673 21025 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 04:02:25.348323 21040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:25.348484 21025 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 04:02:25.348592 21025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790395095474643-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790395095474643-392-0/minicluster-data/master-0/wal
uuid: "1a5584d00a3d41bc9d0a874b861737a3"
format_stamp: "Formatted at 2026-09-26 04:02:25 on dist-test-slave-nhpw"
server_key: "0fbf543ff39d28fd64f36b111ebe8f05"
server_key_iv: "66f8e40556be05276825e5f83bd77311"
server_key_version: "encryptionkey@0"
I20260926 04:02:25.348708 21025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790395095474643-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790395095474643-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790395095474643-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:02:25.369634 21025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:02:25.371698 21025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:02:25.371872 21025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:02:25.375922 21025 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34009
I20260926 04:02:25.375960 21092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34009 every 8 connection(s)
I20260926 04:02:25.376351 21025 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
I20260926 04:02:25.376823 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 21025
I20260926 04:02:25.376953 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790395095474643-392-0/minicluster-data/master-0/wal/instance
I20260926 04:02:25.377146 392 external_mini_cluster.cc:1428] Setting key 25957e15d9b702d74ed9413b3494a52f
I20260926 04:02:25.377414 21093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:02:25.380787 21093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a5584d00a3d41bc9d0a874b861737a3: Bootstrap starting.
I20260926 04:02:25.381469 21093 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a5584d00a3d41bc9d0a874b861737a3: Neither blocks nor log segments found. Creating new log.
I20260926 04:02:25.381829 21093 log.cc:824] T 00000000000000000000000000000000 P 1a5584d00a3d41bc9d0a874b861737a3: Log is configured to *not* fsync() on all Append() calls
I20260926 04:02:25.382673 21093 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a5584d00a3d41bc9d0a874b861737a3: No bootstrap required, opened a new log
I20260926 04:02:25.383258 21096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:25.377641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60359 (local address 127.0.0.1:34009)
0926 04:02:25.377838 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:25.377846 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:25.377882 (+ 36us) server_negotiation.cc:407] Connection header received
0926 04:02:25.378605 (+ 723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:25.378612 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:25.378661 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:25.378778 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:25.379051 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:25.379946 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:25.381128 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:25.381615 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:25.381720 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:25.382559 (+ 839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:25.382563 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:25.382564 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:02:25.382769 (+ 205us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:25.382815 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:02:25.382832 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:25.382902 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:25.382968 (+ 66us) server_negotiation.cc:299] Negotiation successful
0926 04:02:25.383044 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":100,"threads_started":1}
I20260926 04:02:25.384473 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 04:02:25.386723 21093 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a5584d00a3d41bc9d0a874b861737a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a5584d00a3d41bc9d0a874b861737a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34009 } }
I20260926 04:02:25.386901 21093 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a5584d00a3d41bc9d0a874b861737a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:02:25.386933 21093 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a5584d00a3d41bc9d0a874b861737a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a5584d00a3d41bc9d0a874b861737a3, State: Initialized, Role: FOLLOWER
I20260926 04:02:25.387043 21093 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a5584d00a3d41bc9d0a874b861737a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a5584d00a3d41bc9d0a874b861737a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34009 } }
I20260926 04:02:25.387109 21093 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a5584d00a3d41bc9d0a874b861737a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:02:25.387140 21093 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a5584d00a3d41bc9d0a874b861737a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:02:25.387198 21093 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a5584d00a3d41bc9d0a874b861737a3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:02:25.387801 21093 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a5584d00a3d41bc9d0a874b861737a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a5584d00a3d41bc9d0a874b861737a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34009 } }
I20260926 04:02:25.387933 21093 leader_election.cc:302] T 00000000000000000000000000000000 P 1a5584d00a3d41bc9d0a874b861737a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a5584d00a3d41bc9d0a874b861737a3; no voters:
I20260926 04:02:25.388098 21093 leader_election.cc:288] T 00000000000000000000000000000000 P 1a5584d00a3d41bc9d0a874b861737a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 04:02:25.388162 21103 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a5584d00a3d41bc9d0a874b861737a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:02:25.388329 21103 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a5584d00a3d41bc9d0a874b861737a3 [term 1 LEADER]: Becoming Leader. State: Replica: 1a5584d00a3d41bc9d0a874b861737a3, State: Running, Role: LEADER
I20260926 04:02:25.388345 21093 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a5584d00a3d41bc9d0a874b861737a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:02:25.388413 21103 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a5584d00a3d41bc9d0a874b861737a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a5584d00a3d41bc9d0a874b861737a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34009 } }
I20260926 04:02:25.388763 21104 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a5584d00a3d41bc9d0a874b861737a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a5584d00a3d41bc9d0a874b861737a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a5584d00a3d41bc9d0a874b861737a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34009 } } }
I20260926 04:02:25.388866 21104 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a5584d00a3d41bc9d0a874b861737a3 [sys.catalog]: This master's current role is: LEADER
I20260926 04:02:25.388764 21105 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a5584d00a3d41bc9d0a874b861737a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a5584d00a3d41bc9d0a874b861737a3. Latest consensus state: current_term: 1 leader_uuid: "1a5584d00a3d41bc9d0a874b861737a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a5584d00a3d41bc9d0a874b861737a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34009 } } }
I20260926 04:02:25.389009 21105 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a5584d00a3d41bc9d0a874b861737a3 [sys.catalog]: This master's current role is: LEADER
I20260926 04:02:25.389114 21110 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:02:25.389554 21096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:25.386112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38107 (local address 127.0.0.1:34009)
0926 04:02:25.386172 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:25.386174 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:25.386210 (+ 36us) server_negotiation.cc:407] Connection header received
0926 04:02:25.386283 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:25.386284 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:25.386294 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:25.386348 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:25.386441 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:25.387117 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:25.387878 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:25.388196 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:25.389047 (+ 851us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:25.389094 (+ 47us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:25.389095 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:25.389095 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:02:25.389099 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:25.389109 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:02:25.389114 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:25.389402 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:25.389423 (+ 21us) server_negotiation.cc:299] Negotiation successful
0926 04:02:25.389449 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 04:02:25.389638 21110 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:02:25.391371 21110 catalog_manager.cc:1347] Generated new cluster ID: d5e5891b98174e9d9b6ad3ac87c09a04
I20260926 04:02:25.391429 21110 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 04:02:25.394927 21110 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:02:25.395803 21110 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:02:25.404153 21110 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a5584d00a3d41bc9d0a874b861737a3: Generated new TSK 0
I20260926 04:02:25.404345 21110 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 04:02:25.408852 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 21025
2026-09-26T04:02:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1305 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790395095474643-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 04:02:25 dist-test-slave-nhpw krb5kdc[21124](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 04:02:25 dist-test-slave-nhpw krb5kdc[21124](info): set up 2 sockets
Sep 26 04:02:25 dist-test-slave-nhpw krb5kdc[21124](info): commencing operation
krb5kdc: starting...
W20260926 04:02:26.444767 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790395095474643-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790395095474643-392-0/krb5kdc/test-user.keytab.
Sep 26 04:02:26 dist-test-slave-nhpw krb5kdc[21124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395346, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T04:02:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:02:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790395095474643-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790395095474643-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 04:02:26.567051 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790395095474643-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790395095474643-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790395095474643-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33017
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790395095474643-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33017
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:02:26.657661 21137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:02:26.657841 21137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:02:26.657863 21137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:02:26.658998 21137 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:02:26.659040 21137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:02:26.659055 21137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:02:26.659104 21137 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:02:26.659132 21137 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:02:26.660579 21137 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790395095474643-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790395095474643-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33017
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790395095474643-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790395095474643-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:02:26.660825 21137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:02:26.660998 21137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:02:26.665448 21145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:02:26.665475 21142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:02:26.665449 21143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:02:26.665565 21137 server_base.cc:1034] running on GCE node
I20260926 04:02:26.665925 21137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:02:26.666196 21137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:02:26.667394 21137 hybrid_clock.cc:648] HybridClock initialized: now 1790395346667382 us; error 41 us; skew 500 ppm
Sep 26 04:02:26 dist-test-slave-nhpw krb5kdc[21124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395346, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:02:26.669899 21137 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 04:02:26.670347 21137 webserver.cc:460] Webserver started at http://127.0.0.1:35705/ using document root <none> and password file <none>
I20260926 04:02:26.670567 21137 fs_manager.cc:359] Metadata directory not provided
I20260926 04:02:26.670648 21137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:02:26.670801 21137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:02:26.671864 21137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790395095474643-392-0/minicluster-data/master-0/data/instance:
uuid: "964e0185aca346e8b4359546f84876ee"
format_stamp: "Formatted at 2026-09-26 04:02:26 on dist-test-slave-nhpw"
server_key: "47e7d86e98f46d6d14a94713fa955163"
server_key_iv: "e90519dedd8e60a0ed28b5269d0e3112"
server_key_version: "encryptionkey@0"
I20260926 04:02:26.672209 21137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790395095474643-392-0/minicluster-data/master-0/wal/instance:
uuid: "964e0185aca346e8b4359546f84876ee"
format_stamp: "Formatted at 2026-09-26 04:02:26 on dist-test-slave-nhpw"
server_key: "47e7d86e98f46d6d14a94713fa955163"
server_key_iv: "e90519dedd8e60a0ed28b5269d0e3112"
server_key_version: "encryptionkey@0"
I20260926 04:02:26.673571 21137 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 04:02:26.674284 21152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:26.674451 21137 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:26.674546 21137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790395095474643-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790395095474643-392-0/minicluster-data/master-0/wal
uuid: "964e0185aca346e8b4359546f84876ee"
format_stamp: "Formatted at 2026-09-26 04:02:26 on dist-test-slave-nhpw"
server_key: "47e7d86e98f46d6d14a94713fa955163"
server_key_iv: "e90519dedd8e60a0ed28b5269d0e3112"
server_key_version: "encryptionkey@0"
I20260926 04:02:26.674629 21137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790395095474643-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790395095474643-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790395095474643-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:02:26.677891 21137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:02:26.679797 21137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:02:26.679946 21137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:02:26.683936 21137 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33017
I20260926 04:02:26.683967 21204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33017 every 8 connection(s)
I20260926 04:02:26.684273 21137 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
I20260926 04:02:26.685249 21205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:02:26.687284 21205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 964e0185aca346e8b4359546f84876ee: Bootstrap starting.
I20260926 04:02:26.687773 21205 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 964e0185aca346e8b4359546f84876ee: Neither blocks nor log segments found. Creating new log.
I20260926 04:02:26.687995 21205 log.cc:824] T 00000000000000000000000000000000 P 964e0185aca346e8b4359546f84876ee: Log is configured to *not* fsync() on all Append() calls
I20260926 04:02:26.688586 21205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 964e0185aca346e8b4359546f84876ee: No bootstrap required, opened a new log
I20260926 04:02:26.690137 21205 raft_consensus.cc:348] T 00000000000000000000000000000000 P 964e0185aca346e8b4359546f84876ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "964e0185aca346e8b4359546f84876ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33017 } }
I20260926 04:02:26.690266 21205 raft_consensus.cc:374] T 00000000000000000000000000000000 P 964e0185aca346e8b4359546f84876ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:02:26.690289 21205 raft_consensus.cc:729] T 00000000000000000000000000000000 P 964e0185aca346e8b4359546f84876ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 964e0185aca346e8b4359546f84876ee, State: Initialized, Role: FOLLOWER
I20260926 04:02:26.690415 21205 consensus_queue.cc:260] T 00000000000000000000000000000000 P 964e0185aca346e8b4359546f84876ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "964e0185aca346e8b4359546f84876ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33017 } }
I20260926 04:02:26.690491 21205 raft_consensus.cc:388] T 00000000000000000000000000000000 P 964e0185aca346e8b4359546f84876ee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:02:26.690513 21205 raft_consensus.cc:482] T 00000000000000000000000000000000 P 964e0185aca346e8b4359546f84876ee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:02:26.690539 21205 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 964e0185aca346e8b4359546f84876ee [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:02:26.690999 21205 raft_consensus.cc:504] T 00000000000000000000000000000000 P 964e0185aca346e8b4359546f84876ee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "964e0185aca346e8b4359546f84876ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33017 } }
I20260926 04:02:26.691075 21205 leader_election.cc:302] T 00000000000000000000000000000000 P 964e0185aca346e8b4359546f84876ee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 964e0185aca346e8b4359546f84876ee; no voters:
I20260926 04:02:26.691231 21205 leader_election.cc:288] T 00000000000000000000000000000000 P 964e0185aca346e8b4359546f84876ee [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 04:02:26.691259 21208 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 964e0185aca346e8b4359546f84876ee [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:02:26.691377 21208 raft_consensus.cc:686] T 00000000000000000000000000000000 P 964e0185aca346e8b4359546f84876ee [term 1 LEADER]: Becoming Leader. State: Replica: 964e0185aca346e8b4359546f84876ee, State: Running, Role: LEADER
I20260926 04:02:26.691481 21205 sys_catalog.cc:564] T 00000000000000000000000000000000 P 964e0185aca346e8b4359546f84876ee [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:02:26.691514 21208 consensus_queue.cc:237] T 00000000000000000000000000000000 P 964e0185aca346e8b4359546f84876ee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "964e0185aca346e8b4359546f84876ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33017 } }
I20260926 04:02:26.691609 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 21137
I20260926 04:02:26.691730 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790395095474643-392-0/minicluster-data/master-0/wal/instance
I20260926 04:02:26.691924 392 external_mini_cluster.cc:1428] Setting key 6dcdf244b2de47473e836d39d0bf7b49
I20260926 04:02:26.692348 21209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 964e0185aca346e8b4359546f84876ee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "964e0185aca346e8b4359546f84876ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "964e0185aca346e8b4359546f84876ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33017 } } }
I20260926 04:02:26.692444 21209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 964e0185aca346e8b4359546f84876ee [sys.catalog]: This master's current role is: LEADER
I20260926 04:02:26.692663 21220 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:02:26.692782 21210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 964e0185aca346e8b4359546f84876ee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 964e0185aca346e8b4359546f84876ee. Latest consensus state: current_term: 1 leader_uuid: "964e0185aca346e8b4359546f84876ee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "964e0185aca346e8b4359546f84876ee" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33017 } } }
I20260926 04:02:26.692880 21210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 964e0185aca346e8b4359546f84876ee [sys.catalog]: This master's current role is: LEADER
I20260926 04:02:26.693301 21220 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:02:26.694895 21220 catalog_manager.cc:1347] Generated new cluster ID: a7cb8f12077e412f945e403041fae771
I20260926 04:02:26.694943 21220 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 04:02:26.695276 21219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:26.692421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57189 (local address 127.0.0.1:33017)
0926 04:02:26.693301 (+ 880us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:26.693310 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:26.693657 (+ 347us) server_negotiation.cc:407] Connection header received
0926 04:02:26.693856 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:26.693864 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:26.693895 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:26.694051 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:26.694786 (+ 735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:26.694790 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:26.694790 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:02:26.694794 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:26.694835 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:02:26.694850 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:26.694951 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:26.695068 (+ 117us) server_negotiation.cc:299] Negotiation successful
0926 04:02:26.695110 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":827,"thread_start_us":91,"threads_started":1}
I20260926 04:02:26.696100 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 04:02:26.698494 21219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:26.697502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54747 (local address 127.0.0.1:33017)
0926 04:02:26.697534 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:26.697535 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:26.697626 (+ 91us) server_negotiation.cc:407] Connection header received
0926 04:02:26.697674 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:26.697675 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:26.697679 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:26.697746 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:26.698288 (+ 542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:26.698290 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:26.698290 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:02:26.698294 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:26.698309 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:02:26.698314 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:26.698374 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:26.698423 (+ 49us) server_negotiation.cc:299] Negotiation successful
0926 04:02:26.698435 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 04:02:26.710462 21220 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:02:26.710631 21220 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:02:26.713968 21220 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 964e0185aca346e8b4359546f84876ee: Generated new TSK 0
I20260926 04:02:26.714134 21220 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 04:02:26.720518 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 21137
2026-09-26T04:02:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1311 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790395095474643-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 04:02:26 dist-test-slave-nhpw krb5kdc[21236](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 04:02:26 dist-test-slave-nhpw krb5kdc[21236](info): set up 2 sockets
Sep 26 04:02:26 dist-test-slave-nhpw krb5kdc[21236](info): commencing operation
krb5kdc: starting...
W20260926 04:02:27.748773 392 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790395095474643-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790395095474643-392-0/krb5kdc/test-user.keytab.
Sep 26 04:02:27 dist-test-slave-nhpw krb5kdc[21236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T04:02:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:02:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790395095474643-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790395095474643-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 04:02:27.869271 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790395095474643-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790395095474643-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790395095474643-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42873
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790395095474643-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42873
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:02:27.947906 21249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:02:27.948114 21249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:02:27.948138 21249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:02:27.949321 21249 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:02:27.949363 21249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:02:27.949378 21249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:02:27.949389 21249 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:02:27.949401 21249 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:02:27.950865 21249 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790395095474643-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790395095474643-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790395095474643-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21249
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790395095474643-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:02:27.951128 21249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:02:27.951354 21249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:02:27.956171 21254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:02:27.956171 21257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:02:27.956203 21255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:02:27.956597 21249 server_base.cc:1034] running on GCE node
I20260926 04:02:27.956893 21249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:02:27.957185 21249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:02:27.958344 21249 hybrid_clock.cc:648] HybridClock initialized: now 1790395347958332 us; error 28 us; skew 500 ppm
Sep 26 04:02:27 dist-test-slave-nhpw krb5kdc[21236](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395347, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:02:27.960789 21249 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 04:02:27.961193 21249 webserver.cc:460] Webserver started at http://127.0.0.1:46007/ using document root <none> and password file <none>
I20260926 04:02:27.961426 21249 fs_manager.cc:359] Metadata directory not provided
I20260926 04:02:27.961490 21249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:02:27.961632 21249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:02:27.962653 21249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790395095474643-392-0/minicluster-data/master-0/data/instance:
uuid: "62ce48b10b2b47719072af3e92eb95c3"
format_stamp: "Formatted at 2026-09-26 04:02:27 on dist-test-slave-nhpw"
server_key: "194030ae1c736d37745f1d629ad8c112"
server_key_iv: "7920ef3f3af0b5242353e429626f0c98"
server_key_version: "encryptionkey@0"
I20260926 04:02:27.963006 21249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790395095474643-392-0/minicluster-data/master-0/wal/instance:
uuid: "62ce48b10b2b47719072af3e92eb95c3"
format_stamp: "Formatted at 2026-09-26 04:02:27 on dist-test-slave-nhpw"
server_key: "194030ae1c736d37745f1d629ad8c112"
server_key_iv: "7920ef3f3af0b5242353e429626f0c98"
server_key_version: "encryptionkey@0"
I20260926 04:02:27.964287 21249 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 04:02:27.965060 21264 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:27.965231 21249 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:27.965327 21249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790395095474643-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790395095474643-392-0/minicluster-data/master-0/wal
uuid: "62ce48b10b2b47719072af3e92eb95c3"
format_stamp: "Formatted at 2026-09-26 04:02:27 on dist-test-slave-nhpw"
server_key: "194030ae1c736d37745f1d629ad8c112"
server_key_iv: "7920ef3f3af0b5242353e429626f0c98"
server_key_version: "encryptionkey@0"
I20260926 04:02:27.965406 21249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790395095474643-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790395095474643-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790395095474643-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:02:27.976991 21249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:02:27.979285 21249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:02:27.979444 21249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:02:27.983625 21249 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42873
I20260926 04:02:27.983661 21316 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42873 every 8 connection(s)
I20260926 04:02:27.983995 21249 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
I20260926 04:02:27.984915 21317 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:02:27.986994 21317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62ce48b10b2b47719072af3e92eb95c3: Bootstrap starting.
I20260926 04:02:27.987485 21317 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62ce48b10b2b47719072af3e92eb95c3: Neither blocks nor log segments found. Creating new log.
I20260926 04:02:27.987720 21317 log.cc:824] T 00000000000000000000000000000000 P 62ce48b10b2b47719072af3e92eb95c3: Log is configured to *not* fsync() on all Append() calls
I20260926 04:02:27.988348 21317 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62ce48b10b2b47719072af3e92eb95c3: No bootstrap required, opened a new log
I20260926 04:02:27.989912 21317 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62ce48b10b2b47719072af3e92eb95c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62ce48b10b2b47719072af3e92eb95c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42873 } }
I20260926 04:02:27.990041 21317 raft_consensus.cc:374] T 00000000000000000000000000000000 P 62ce48b10b2b47719072af3e92eb95c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:02:27.990063 21317 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62ce48b10b2b47719072af3e92eb95c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62ce48b10b2b47719072af3e92eb95c3, State: Initialized, Role: FOLLOWER
I20260926 04:02:27.990190 21317 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62ce48b10b2b47719072af3e92eb95c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62ce48b10b2b47719072af3e92eb95c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42873 } }
I20260926 04:02:27.990265 21317 raft_consensus.cc:388] T 00000000000000000000000000000000 P 62ce48b10b2b47719072af3e92eb95c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:02:27.990288 21317 raft_consensus.cc:482] T 00000000000000000000000000000000 P 62ce48b10b2b47719072af3e92eb95c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:02:27.990315 21317 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62ce48b10b2b47719072af3e92eb95c3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:02:27.990774 21317 raft_consensus.cc:504] T 00000000000000000000000000000000 P 62ce48b10b2b47719072af3e92eb95c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62ce48b10b2b47719072af3e92eb95c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42873 } }
I20260926 04:02:27.990854 21317 leader_election.cc:302] T 00000000000000000000000000000000 P 62ce48b10b2b47719072af3e92eb95c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 62ce48b10b2b47719072af3e92eb95c3; no voters:
I20260926 04:02:27.990965 21317 leader_election.cc:288] T 00000000000000000000000000000000 P 62ce48b10b2b47719072af3e92eb95c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 04:02:27.991036 21320 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 62ce48b10b2b47719072af3e92eb95c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:02:27.991204 21320 raft_consensus.cc:686] T 00000000000000000000000000000000 P 62ce48b10b2b47719072af3e92eb95c3 [term 1 LEADER]: Becoming Leader. State: Replica: 62ce48b10b2b47719072af3e92eb95c3, State: Running, Role: LEADER
I20260926 04:02:27.991225 21317 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62ce48b10b2b47719072af3e92eb95c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:02:27.991321 21320 consensus_queue.cc:237] T 00000000000000000000000000000000 P 62ce48b10b2b47719072af3e92eb95c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62ce48b10b2b47719072af3e92eb95c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42873 } }
I20260926 04:02:27.991667 21322 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62ce48b10b2b47719072af3e92eb95c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 62ce48b10b2b47719072af3e92eb95c3. Latest consensus state: current_term: 1 leader_uuid: "62ce48b10b2b47719072af3e92eb95c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62ce48b10b2b47719072af3e92eb95c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42873 } } }
I20260926 04:02:27.991783 21321 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62ce48b10b2b47719072af3e92eb95c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "62ce48b10b2b47719072af3e92eb95c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "62ce48b10b2b47719072af3e92eb95c3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42873 } } }
I20260926 04:02:27.991803 21322 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62ce48b10b2b47719072af3e92eb95c3 [sys.catalog]: This master's current role is: LEADER
I20260926 04:02:27.991871 21321 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62ce48b10b2b47719072af3e92eb95c3 [sys.catalog]: This master's current role is: LEADER
I20260926 04:02:27.992296 21329 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:02:27.992851 21329 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:02:27.993714 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 21249
I20260926 04:02:27.993845 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790395095474643-392-0/minicluster-data/master-0/wal/instance
I20260926 04:02:27.994086 392 external_mini_cluster.cc:1428] Setting key 336a1a843659471d5e753748b0f2eb38
I20260926 04:02:27.994408 21329 catalog_manager.cc:1347] Generated new cluster ID: 2dfe96c3ac4a460991002be9ffe3a89d
I20260926 04:02:27.994453 21329 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 04:02:27 dist-test-slave-nhpw krb5kdc[21236](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395347, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 04:02:28.005492 21340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:27.994624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.26:58209 (local address 127.0.0.1:42873)
0926 04:02:27.996868 (+ 2244us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:27.996874 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:27.996897 (+ 23us) server_negotiation.cc:407] Connection header received
0926 04:02:27.996978 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:27.996983 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:27.997033 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:27.997186 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:27.997784 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:27.998721 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:27.999479 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:27.999888 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:28.000004 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:28.002296 (+ 2292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:28.002299 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:28.002300 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:02:28.002327 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:28.003937 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:28.004331 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:28.004334 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:28.004335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:28.004420 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:28.004698 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:28.004699 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:28.004699 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:28.004910 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:28.005113 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:28.005195 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 04:02:28.005294 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2206,"thread_start_us":70,"threads_started":1}
I20260926 04:02:28.006430 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 04:02:28.019034 21340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:28.012968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.26:47215 (local address 127.0.0.1:42873)
0926 04:02:28.013007 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:28.013008 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:28.013149 (+ 141us) server_negotiation.cc:407] Connection header received
0926 04:02:28.013215 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:28.013216 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:28.013228 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:28.013301 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:28.013805 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:28.014290 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:28.014973 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:28.015191 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:28.015269 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:28.016413 (+ 1144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:28.016415 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:28.016415 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:02:28.016422 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:28.017838 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:28.018227 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:28.018233 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:28.018233 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:28.018280 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:28.018499 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:28.018500 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:28.018500 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:28.018619 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:28.018754 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:28.018859 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 04:02:28.018919 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 04:02:28.019676 21329 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:02:28.020339 21329 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:02:28.026548 21329 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 62ce48b10b2b47719072af3e92eb95c3: Generated new TSK 0
I20260926 04:02:28.026733 21329 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 04:02:28.031492 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 21249
W20260926 04:02:28.035585 21234 connection.cc:535] client connection to 127.0.0.1:42873 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42873): Connection reset by peer (error 104)
2026-09-26T04:02:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1310 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790395095474643-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 04:02:28 dist-test-slave-nhpw krb5kdc[21348](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 04:02:28 dist-test-slave-nhpw krb5kdc[21348](info): set up 2 sockets
Sep 26 04:02:28 dist-test-slave-nhpw krb5kdc[21348](info): commencing operation
krb5kdc: starting...
W20260926 04:02:29.052832 392 mini_kdc.cc:121] Time spent starting KDC: real 1.004s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790395095474643-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790395095474643-392-0/krb5kdc/test-user.keytab.
Sep 26 04:02:29 dist-test-slave-nhpw krb5kdc[21348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395349, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T04:02:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:02:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790395095474643-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790395095474643-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 04:02:29.171116 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790395095474643-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790395095474643-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790395095474643-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43485
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790395095474643-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43485
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:02:29.251598 21361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:02:29.251809 21361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:02:29.251833 21361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:02:29.253099 21361 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:02:29.253146 21361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:02:29.253161 21361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:02:29.253173 21361 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:02:29.253186 21361 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:02:29.254680 21361 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790395095474643-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790395095474643-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43485
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790395095474643-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21361
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790395095474643-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:02:29.254931 21361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:02:29.255196 21361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:02:29.260238 21369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:02:29.260236 21367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:02:29.260509 21361 server_base.cc:1034] running on GCE node
W20260926 04:02:29.260236 21366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:02:29.260833 21361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:02:29.261089 21361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:02:29.262276 21361 hybrid_clock.cc:648] HybridClock initialized: now 1790395349262248 us; error 41 us; skew 500 ppm
Sep 26 04:02:29 dist-test-slave-nhpw krb5kdc[21348](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395349, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:02:29.264840 21361 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 04:02:29.265277 21361 webserver.cc:460] Webserver started at http://127.0.0.1:43031/ using document root <none> and password file <none>
I20260926 04:02:29.265512 21361 fs_manager.cc:359] Metadata directory not provided
I20260926 04:02:29.265558 21361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:02:29.265650 21361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:02:29.266708 21361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790395095474643-392-0/minicluster-data/master-0/data/instance:
uuid: "e3c2678bcd7b4b1cbdd9f1a32136b27c"
format_stamp: "Formatted at 2026-09-26 04:02:29 on dist-test-slave-nhpw"
server_key: "6ee51e75cc14e1acbf04d833fd7b19cc"
server_key_iv: "27910e0fa34db07b3b70821e57b95611"
server_key_version: "encryptionkey@0"
I20260926 04:02:29.267010 21361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790395095474643-392-0/minicluster-data/master-0/wal/instance:
uuid: "e3c2678bcd7b4b1cbdd9f1a32136b27c"
format_stamp: "Formatted at 2026-09-26 04:02:29 on dist-test-slave-nhpw"
server_key: "6ee51e75cc14e1acbf04d833fd7b19cc"
server_key_iv: "27910e0fa34db07b3b70821e57b95611"
server_key_version: "encryptionkey@0"
I20260926 04:02:29.268361 21361 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 04:02:29.269178 21376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:29.269358 21361 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:29.269465 21361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790395095474643-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790395095474643-392-0/minicluster-data/master-0/wal
uuid: "e3c2678bcd7b4b1cbdd9f1a32136b27c"
format_stamp: "Formatted at 2026-09-26 04:02:29 on dist-test-slave-nhpw"
server_key: "6ee51e75cc14e1acbf04d833fd7b19cc"
server_key_iv: "27910e0fa34db07b3b70821e57b95611"
server_key_version: "encryptionkey@0"
I20260926 04:02:29.269544 21361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790395095474643-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790395095474643-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790395095474643-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:02:29.282163 21361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:02:29.284179 21361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:02:29.284343 21361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:02:29.288157 21361 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43485
I20260926 04:02:29.288204 21428 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43485 every 8 connection(s)
I20260926 04:02:29.288820 21361 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
I20260926 04:02:29.289484 21429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:02:29.291685 21429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3c2678bcd7b4b1cbdd9f1a32136b27c: Bootstrap starting.
I20260926 04:02:29.292203 21429 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e3c2678bcd7b4b1cbdd9f1a32136b27c: Neither blocks nor log segments found. Creating new log.
I20260926 04:02:29.292447 21429 log.cc:824] T 00000000000000000000000000000000 P e3c2678bcd7b4b1cbdd9f1a32136b27c: Log is configured to *not* fsync() on all Append() calls
I20260926 04:02:29.293143 21429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e3c2678bcd7b4b1cbdd9f1a32136b27c: No bootstrap required, opened a new log
I20260926 04:02:29.294657 21429 raft_consensus.cc:348] T 00000000000000000000000000000000 P e3c2678bcd7b4b1cbdd9f1a32136b27c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3c2678bcd7b4b1cbdd9f1a32136b27c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43485 } }
I20260926 04:02:29.294804 21429 raft_consensus.cc:374] T 00000000000000000000000000000000 P e3c2678bcd7b4b1cbdd9f1a32136b27c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:02:29.294829 21429 raft_consensus.cc:729] T 00000000000000000000000000000000 P e3c2678bcd7b4b1cbdd9f1a32136b27c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3c2678bcd7b4b1cbdd9f1a32136b27c, State: Initialized, Role: FOLLOWER
I20260926 04:02:29.294974 21429 consensus_queue.cc:260] T 00000000000000000000000000000000 P e3c2678bcd7b4b1cbdd9f1a32136b27c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3c2678bcd7b4b1cbdd9f1a32136b27c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43485 } }
I20260926 04:02:29.295056 21429 raft_consensus.cc:388] T 00000000000000000000000000000000 P e3c2678bcd7b4b1cbdd9f1a32136b27c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:02:29.295079 21429 raft_consensus.cc:482] T 00000000000000000000000000000000 P e3c2678bcd7b4b1cbdd9f1a32136b27c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:02:29.295107 21429 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e3c2678bcd7b4b1cbdd9f1a32136b27c [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:02:29.295835 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 21361
I20260926 04:02:29.295956 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790395095474643-392-0/minicluster-data/master-0/wal/instance
I20260926 04:02:29.296175 392 external_mini_cluster.cc:1428] Setting key 44cf345fe63ecb86952ef219d75133e6
I20260926 04:02:29.296209 21429 raft_consensus.cc:504] T 00000000000000000000000000000000 P e3c2678bcd7b4b1cbdd9f1a32136b27c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3c2678bcd7b4b1cbdd9f1a32136b27c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43485 } }
I20260926 04:02:29.296329 21429 leader_election.cc:302] T 00000000000000000000000000000000 P e3c2678bcd7b4b1cbdd9f1a32136b27c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e3c2678bcd7b4b1cbdd9f1a32136b27c; no voters:
I20260926 04:02:29.296522 21429 leader_election.cc:288] T 00000000000000000000000000000000 P e3c2678bcd7b4b1cbdd9f1a32136b27c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 04:02:29.296564 21432 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e3c2678bcd7b4b1cbdd9f1a32136b27c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:02:29.296873 21429 sys_catalog.cc:564] T 00000000000000000000000000000000 P e3c2678bcd7b4b1cbdd9f1a32136b27c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:02:29.297004 21432 raft_consensus.cc:686] T 00000000000000000000000000000000 P e3c2678bcd7b4b1cbdd9f1a32136b27c [term 1 LEADER]: Becoming Leader. State: Replica: e3c2678bcd7b4b1cbdd9f1a32136b27c, State: Running, Role: LEADER
I20260926 04:02:29.297168 21432 consensus_queue.cc:237] T 00000000000000000000000000000000 P e3c2678bcd7b4b1cbdd9f1a32136b27c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3c2678bcd7b4b1cbdd9f1a32136b27c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43485 } }
I20260926 04:02:29.297559 21432 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3c2678bcd7b4b1cbdd9f1a32136b27c [sys.catalog]: SysCatalogTable state changed. Reason: New leader e3c2678bcd7b4b1cbdd9f1a32136b27c. Latest consensus state: current_term: 1 leader_uuid: "e3c2678bcd7b4b1cbdd9f1a32136b27c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3c2678bcd7b4b1cbdd9f1a32136b27c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43485 } } }
I20260926 04:02:29.297569 21436 sys_catalog.cc:455] T 00000000000000000000000000000000 P e3c2678bcd7b4b1cbdd9f1a32136b27c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e3c2678bcd7b4b1cbdd9f1a32136b27c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e3c2678bcd7b4b1cbdd9f1a32136b27c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43485 } } }
I20260926 04:02:29.297717 21432 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3c2678bcd7b4b1cbdd9f1a32136b27c [sys.catalog]: This master's current role is: LEADER
I20260926 04:02:29.297767 21436 sys_catalog.cc:458] T 00000000000000000000000000000000 P e3c2678bcd7b4b1cbdd9f1a32136b27c [sys.catalog]: This master's current role is: LEADER
I20260926 04:02:29.298161 21444 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:02:29.298688 21444 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:02:29.300765 21444 catalog_manager.cc:1347] Generated new cluster ID: 9d2b19e2ca454db28388ac14278b32a5
I20260926 04:02:29.300819 21444 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 04:02:29.301911 21435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:29.296731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.26:51553 (local address 127.0.0.1:43485)
0926 04:02:29.296908 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:29.296918 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:29.296954 (+ 36us) server_negotiation.cc:407] Connection header received
0926 04:02:29.297106 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:29.297113 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:29.297173 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:29.297297 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:29.297519 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:29.298942 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:29.299816 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:29.300334 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:29.300457 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:29.301282 (+ 825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:29.301286 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:29.301286 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:02:29.301477 (+ 191us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:29.301508 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:02:29.301519 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:29.301617 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:29.301696 (+ 79us) server_negotiation.cc:299] Negotiation successful
0926 04:02:29.301732 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":63,"threads_started":1}
I20260926 04:02:29.305845 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 04:02:29.315873 21435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:29.308941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.26:34997 (local address 127.0.0.1:43485)
0926 04:02:29.308983 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:29.308985 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:29.312867 (+ 3882us) server_negotiation.cc:407] Connection header received
0926 04:02:29.312984 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:29.312985 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:29.312999 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:29.313028 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:29.313243 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:29.313756 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:29.314537 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:29.314767 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:29.314857 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:29.315577 (+ 720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:29.315578 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:29.315578 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:02:29.315583 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:29.315597 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:02:29.315604 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:29.315676 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:29.315755 (+ 79us) server_negotiation.cc:299] Negotiation successful
0926 04:02:29.315767 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 04:02:29.356988 21444 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:02:29.357558 21444 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:02:29.365541 21444 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e3c2678bcd7b4b1cbdd9f1a32136b27c: Generated new TSK 0
I20260926 04:02:29.365751 21444 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 04:02:29.370079 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 21361
W20260926 04:02:29.374294 21346 connection.cc:535] client connection to 127.0.0.1:43485 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43485): Connection reset by peer (error 104)
2026-09-26T04:02:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1339 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790395095474643-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 04:02:29 dist-test-slave-nhpw krb5kdc[21460](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 04:02:29 dist-test-slave-nhpw krb5kdc[21460](info): set up 2 sockets
Sep 26 04:02:29 dist-test-slave-nhpw krb5kdc[21460](info): commencing operation
krb5kdc: starting...
W20260926 04:02:30.404841 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790395095474643-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790395095474643-392-0/krb5kdc/test-user.keytab.
Sep 26 04:02:30 dist-test-slave-nhpw krb5kdc[21460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T04:02:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:02:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790395095474643-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790395095474643-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 04:02:30.519320 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790395095474643-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790395095474643-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790395095474643-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33319
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790395095474643-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33319
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:02:30.596735 21473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:02:30.596926 21473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:02:30.596949 21473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:02:30.598029 21473 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:02:30.598065 21473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:02:30.598079 21473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:02:30.598093 21473 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:02:30.598104 21473 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:02:30.599465 21473 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790395095474643-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790395095474643-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33319
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790395095474643-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790395095474643-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:02:30.599649 21473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:02:30.599815 21473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:02:30.604455 21479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:02:30.604450 21481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:02:30.604451 21478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:02:30.604799 21473 server_base.cc:1034] running on GCE node
I20260926 04:02:30.605055 21473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:02:30.605342 21473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:02:30.606520 21473 hybrid_clock.cc:648] HybridClock initialized: now 1790395350606500 us; error 36 us; skew 500 ppm
Sep 26 04:02:30 dist-test-slave-nhpw krb5kdc[21460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395350, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:02:30.609236 21473 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 04:02:30.609614 21473 webserver.cc:460] Webserver started at http://127.0.0.1:43953/ using document root <none> and password file <none>
I20260926 04:02:30.609798 21473 fs_manager.cc:359] Metadata directory not provided
I20260926 04:02:30.609838 21473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:02:30.609920 21473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:02:30.610903 21473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790395095474643-392-0/minicluster-data/master-0/data/instance:
uuid: "a31866c7d9e74ca0a5dc60a76380eabb"
format_stamp: "Formatted at 2026-09-26 04:02:30 on dist-test-slave-nhpw"
server_key: "d769e5b79050703363f5ede3b42a00e3"
server_key_iv: "53beb1f12656f359a80a3dd59e91b5a3"
server_key_version: "encryptionkey@0"
I20260926 04:02:30.611197 21473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790395095474643-392-0/minicluster-data/master-0/wal/instance:
uuid: "a31866c7d9e74ca0a5dc60a76380eabb"
format_stamp: "Formatted at 2026-09-26 04:02:30 on dist-test-slave-nhpw"
server_key: "d769e5b79050703363f5ede3b42a00e3"
server_key_iv: "53beb1f12656f359a80a3dd59e91b5a3"
server_key_version: "encryptionkey@0"
I20260926 04:02:30.612439 21473 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 04:02:30.613196 21488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:30.613353 21473 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:30.613420 21473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790395095474643-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790395095474643-392-0/minicluster-data/master-0/wal
uuid: "a31866c7d9e74ca0a5dc60a76380eabb"
format_stamp: "Formatted at 2026-09-26 04:02:30 on dist-test-slave-nhpw"
server_key: "d769e5b79050703363f5ede3b42a00e3"
server_key_iv: "53beb1f12656f359a80a3dd59e91b5a3"
server_key_version: "encryptionkey@0"
I20260926 04:02:30.613473 21473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790395095474643-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790395095474643-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790395095474643-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:02:30.616861 21473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:02:30.619084 21473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:02:30.619190 21473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:02:30.623412 21473 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33319
I20260926 04:02:30.623903 21473 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
I20260926 04:02:30.624817 21540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33319 every 8 connection(s)
I20260926 04:02:30.625146 21541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:02:30.628556 21541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a31866c7d9e74ca0a5dc60a76380eabb: Bootstrap starting.
I20260926 04:02:30.629290 21541 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a31866c7d9e74ca0a5dc60a76380eabb: Neither blocks nor log segments found. Creating new log.
I20260926 04:02:30.629624 21541 log.cc:824] T 00000000000000000000000000000000 P a31866c7d9e74ca0a5dc60a76380eabb: Log is configured to *not* fsync() on all Append() calls
I20260926 04:02:30.630319 21541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a31866c7d9e74ca0a5dc60a76380eabb: No bootstrap required, opened a new log
I20260926 04:02:30.632591 21541 raft_consensus.cc:348] T 00000000000000000000000000000000 P a31866c7d9e74ca0a5dc60a76380eabb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a31866c7d9e74ca0a5dc60a76380eabb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33319 } }
I20260926 04:02:30.632812 21541 raft_consensus.cc:374] T 00000000000000000000000000000000 P a31866c7d9e74ca0a5dc60a76380eabb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:02:30.632860 21541 raft_consensus.cc:729] T 00000000000000000000000000000000 P a31866c7d9e74ca0a5dc60a76380eabb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a31866c7d9e74ca0a5dc60a76380eabb, State: Initialized, Role: FOLLOWER
I20260926 04:02:30.633003 21541 consensus_queue.cc:260] T 00000000000000000000000000000000 P a31866c7d9e74ca0a5dc60a76380eabb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a31866c7d9e74ca0a5dc60a76380eabb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33319 } }
I20260926 04:02:30.633091 21541 raft_consensus.cc:388] T 00000000000000000000000000000000 P a31866c7d9e74ca0a5dc60a76380eabb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:02:30.633129 21541 raft_consensus.cc:482] T 00000000000000000000000000000000 P a31866c7d9e74ca0a5dc60a76380eabb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:02:30.633178 21541 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a31866c7d9e74ca0a5dc60a76380eabb [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:02:30.633324 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 21473
I20260926 04:02:30.633451 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790395095474643-392-0/minicluster-data/master-0/wal/instance
I20260926 04:02:30.633628 392 external_mini_cluster.cc:1428] Setting key fd43cf9dba7a5a1949dfc7c99e002ac9
I20260926 04:02:30.633852 21541 raft_consensus.cc:504] T 00000000000000000000000000000000 P a31866c7d9e74ca0a5dc60a76380eabb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a31866c7d9e74ca0a5dc60a76380eabb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33319 } }
I20260926 04:02:30.633968 21541 leader_election.cc:302] T 00000000000000000000000000000000 P a31866c7d9e74ca0a5dc60a76380eabb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a31866c7d9e74ca0a5dc60a76380eabb; no voters:
I20260926 04:02:30.634124 21541 leader_election.cc:288] T 00000000000000000000000000000000 P a31866c7d9e74ca0a5dc60a76380eabb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 04:02:30.634173 21545 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a31866c7d9e74ca0a5dc60a76380eabb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:02:30.634311 21545 raft_consensus.cc:686] T 00000000000000000000000000000000 P a31866c7d9e74ca0a5dc60a76380eabb [term 1 LEADER]: Becoming Leader. State: Replica: a31866c7d9e74ca0a5dc60a76380eabb, State: Running, Role: LEADER
I20260926 04:02:30.634373 21541 sys_catalog.cc:564] T 00000000000000000000000000000000 P a31866c7d9e74ca0a5dc60a76380eabb [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:02:30.634400 21545 consensus_queue.cc:237] T 00000000000000000000000000000000 P a31866c7d9e74ca0a5dc60a76380eabb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a31866c7d9e74ca0a5dc60a76380eabb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33319 } }
I20260926 04:02:30.634714 21545 sys_catalog.cc:455] T 00000000000000000000000000000000 P a31866c7d9e74ca0a5dc60a76380eabb [sys.catalog]: SysCatalogTable state changed. Reason: New leader a31866c7d9e74ca0a5dc60a76380eabb. Latest consensus state: current_term: 1 leader_uuid: "a31866c7d9e74ca0a5dc60a76380eabb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a31866c7d9e74ca0a5dc60a76380eabb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33319 } } }
I20260926 04:02:30.634802 21545 sys_catalog.cc:458] T 00000000000000000000000000000000 P a31866c7d9e74ca0a5dc60a76380eabb [sys.catalog]: This master's current role is: LEADER
I20260926 04:02:30.634817 21548 sys_catalog.cc:455] T 00000000000000000000000000000000 P a31866c7d9e74ca0a5dc60a76380eabb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a31866c7d9e74ca0a5dc60a76380eabb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a31866c7d9e74ca0a5dc60a76380eabb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33319 } } }
I20260926 04:02:30.634878 21548 sys_catalog.cc:458] T 00000000000000000000000000000000 P a31866c7d9e74ca0a5dc60a76380eabb [sys.catalog]: This master's current role is: LEADER
I20260926 04:02:30.634986 21552 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:02:30.635488 21552 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:02:30.637166 21552 catalog_manager.cc:1347] Generated new cluster ID: 1ba6c4cb464b45448733683f75669c95
I20260926 04:02:30.637269 21552 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 04:02:30.637584 21546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:30.634092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.26:50163 (local address 127.0.0.1:33319)
0926 04:02:30.635988 (+ 1896us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:30.635996 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:30.636020 (+ 24us) server_negotiation.cc:407] Connection header received
0926 04:02:30.636144 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:30.636150 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:30.636196 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:30.636346 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:30.637091 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:30.637095 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:30.637095 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:02:30.637099 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:30.637144 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:02:30.637158 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:30.637221 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:30.637351 (+ 130us) server_negotiation.cc:299] Negotiation successful
0926 04:02:30.637396 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1855,"thread_start_us":62,"threads_started":1}
I20260926 04:02:30.638422 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 04:02:30.640779 21546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:30.639817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.26:36823 (local address 127.0.0.1:33319)
0926 04:02:30.639848 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:30.639849 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:30.639857 (+ 8us) server_negotiation.cc:407] Connection header received
0926 04:02:30.639921 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:30.639921 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:30.639933 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:30.639981 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:30.640554 (+ 573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:30.640555 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:30.640556 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:02:30.640559 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:30.640577 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:02:30.640581 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:30.640652 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:30.640709 (+ 57us) server_negotiation.cc:299] Negotiation successful
0926 04:02:30.640721 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 04:02:30.651372 21552 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:02:30.651551 21552 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:02:30.662180 21552 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a31866c7d9e74ca0a5dc60a76380eabb: Generated new TSK 0
I20260926 04:02:30.662382 21552 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 04:02:30.669188 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 21473
2026-09-26T04:02:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1298 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790395095474643-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 04:02:30 dist-test-slave-nhpw krb5kdc[21572](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 04:02:30 dist-test-slave-nhpw krb5kdc[21572](info): set up 2 sockets
Sep 26 04:02:30 dist-test-slave-nhpw krb5kdc[21572](info): commencing operation
krb5kdc: starting...
W20260926 04:02:31.744827 392 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790395095474643-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790395095474643-392-0/krb5kdc/test-user.keytab.
Sep 26 04:02:31 dist-test-slave-nhpw krb5kdc[21572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395351, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T04:02:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:02:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790395095474643-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790395095474643-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 04:02:31.859778 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790395095474643-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790395095474643-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790395095474643-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42663
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790395095474643-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42663
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:02:31.940101 21585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:02:31.940301 21585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:02:31.940323 21585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:02:31.941394 21585 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:02:31.941428 21585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:02:31.941442 21585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:02:31.941454 21585 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:02:31.941466 21585 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:02:31.942812 21585 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790395095474643-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790395095474643-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42663
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790395095474643-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790395095474643-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:02:31.942984 21585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:02:31.943148 21585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:02:31.947993 21593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:02:31.948038 21591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:02:31.948012 21590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:02:31.948350 21585 server_base.cc:1034] running on GCE node
I20260926 04:02:31.948571 21585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:02:31.948848 21585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:02:31.949994 21585 hybrid_clock.cc:648] HybridClock initialized: now 1790395351949978 us; error 35 us; skew 500 ppm
Sep 26 04:02:31 dist-test-slave-nhpw krb5kdc[21572](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395351, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:02:31.963371 21585 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 04:02:31.963847 21585 webserver.cc:460] Webserver started at http://127.0.0.1:35409/ using document root <none> and password file <none>
I20260926 04:02:31.964066 21585 fs_manager.cc:359] Metadata directory not provided
I20260926 04:02:31.964134 21585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:02:31.964288 21585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:02:31.965382 21585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790395095474643-392-0/minicluster-data/master-0/data/instance:
uuid: "7248acc1c5cf4d7d8ccac6cd34d43c13"
format_stamp: "Formatted at 2026-09-26 04:02:31 on dist-test-slave-nhpw"
server_key: "8c42d8c6bbbf5a9b86410de64c0cb253"
server_key_iv: "d739fb255a140c92111ae4f444d2b121"
server_key_version: "encryptionkey@0"
I20260926 04:02:31.965739 21585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790395095474643-392-0/minicluster-data/master-0/wal/instance:
uuid: "7248acc1c5cf4d7d8ccac6cd34d43c13"
format_stamp: "Formatted at 2026-09-26 04:02:31 on dist-test-slave-nhpw"
server_key: "8c42d8c6bbbf5a9b86410de64c0cb253"
server_key_iv: "d739fb255a140c92111ae4f444d2b121"
server_key_version: "encryptionkey@0"
I20260926 04:02:31.967027 21585 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 04:02:31.967768 21600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:31.967939 21585 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 04:02:31.968045 21585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790395095474643-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790395095474643-392-0/minicluster-data/master-0/wal
uuid: "7248acc1c5cf4d7d8ccac6cd34d43c13"
format_stamp: "Formatted at 2026-09-26 04:02:31 on dist-test-slave-nhpw"
server_key: "8c42d8c6bbbf5a9b86410de64c0cb253"
server_key_iv: "d739fb255a140c92111ae4f444d2b121"
server_key_version: "encryptionkey@0"
I20260926 04:02:31.968120 21585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790395095474643-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790395095474643-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790395095474643-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:02:31.978171 21585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:02:31.980058 21585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:02:31.980228 21585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:02:31.984390 21585 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42663
I20260926 04:02:31.984457 21652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42663 every 8 connection(s)
I20260926 04:02:31.984835 21585 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
I20260926 04:02:31.985749 21653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:02:31.988117 21653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7248acc1c5cf4d7d8ccac6cd34d43c13: Bootstrap starting.
I20260926 04:02:31.988751 21653 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7248acc1c5cf4d7d8ccac6cd34d43c13: Neither blocks nor log segments found. Creating new log.
I20260926 04:02:31.989033 21653 log.cc:824] T 00000000000000000000000000000000 P 7248acc1c5cf4d7d8ccac6cd34d43c13: Log is configured to *not* fsync() on all Append() calls
I20260926 04:02:31.989696 21653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7248acc1c5cf4d7d8ccac6cd34d43c13: No bootstrap required, opened a new log
I20260926 04:02:31.991266 21653 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7248acc1c5cf4d7d8ccac6cd34d43c13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7248acc1c5cf4d7d8ccac6cd34d43c13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42663 } }
I20260926 04:02:31.991406 21653 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7248acc1c5cf4d7d8ccac6cd34d43c13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:02:31.991429 21653 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7248acc1c5cf4d7d8ccac6cd34d43c13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7248acc1c5cf4d7d8ccac6cd34d43c13, State: Initialized, Role: FOLLOWER
I20260926 04:02:31.991602 21653 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7248acc1c5cf4d7d8ccac6cd34d43c13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7248acc1c5cf4d7d8ccac6cd34d43c13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42663 } }
I20260926 04:02:31.991663 21653 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7248acc1c5cf4d7d8ccac6cd34d43c13 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:02:31.991705 21653 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7248acc1c5cf4d7d8ccac6cd34d43c13 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:02:31.991763 21653 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7248acc1c5cf4d7d8ccac6cd34d43c13 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:02:31.992280 21653 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7248acc1c5cf4d7d8ccac6cd34d43c13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7248acc1c5cf4d7d8ccac6cd34d43c13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42663 } }
I20260926 04:02:31.992389 21653 leader_election.cc:302] T 00000000000000000000000000000000 P 7248acc1c5cf4d7d8ccac6cd34d43c13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7248acc1c5cf4d7d8ccac6cd34d43c13; no voters:
I20260926 04:02:31.992570 21653 leader_election.cc:288] T 00000000000000000000000000000000 P 7248acc1c5cf4d7d8ccac6cd34d43c13 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 04:02:31.992602 21656 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7248acc1c5cf4d7d8ccac6cd34d43c13 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:02:31.992839 21653 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7248acc1c5cf4d7d8ccac6cd34d43c13 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:02:31.992918 21656 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7248acc1c5cf4d7d8ccac6cd34d43c13 [term 1 LEADER]: Becoming Leader. State: Replica: 7248acc1c5cf4d7d8ccac6cd34d43c13, State: Running, Role: LEADER
I20260926 04:02:31.993019 21656 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7248acc1c5cf4d7d8ccac6cd34d43c13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7248acc1c5cf4d7d8ccac6cd34d43c13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42663 } }
I20260926 04:02:31.993677 21658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7248acc1c5cf4d7d8ccac6cd34d43c13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7248acc1c5cf4d7d8ccac6cd34d43c13. Latest consensus state: current_term: 1 leader_uuid: "7248acc1c5cf4d7d8ccac6cd34d43c13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7248acc1c5cf4d7d8ccac6cd34d43c13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42663 } } }
I20260926 04:02:31.993755 21658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7248acc1c5cf4d7d8ccac6cd34d43c13 [sys.catalog]: This master's current role is: LEADER
I20260926 04:02:31.993851 21657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7248acc1c5cf4d7d8ccac6cd34d43c13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7248acc1c5cf4d7d8ccac6cd34d43c13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7248acc1c5cf4d7d8ccac6cd34d43c13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42663 } } }
I20260926 04:02:31.993937 21657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7248acc1c5cf4d7d8ccac6cd34d43c13 [sys.catalog]: This master's current role is: LEADER
I20260926 04:02:31.994295 21665 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:02:31.994539 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 21585
I20260926 04:02:31.994671 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790395095474643-392-0/minicluster-data/master-0/wal/instance
I20260926 04:02:31.994851 392 external_mini_cluster.cc:1428] Setting key a668f2ec919570b1ac6b27cc66269879
I20260926 04:02:31.994860 21665 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:02:31.996292 21665 catalog_manager.cc:1347] Generated new cluster ID: 2488cc331dcb4a6a947ed3fdc9a2c176
I20260926 04:02:31.996343 21665 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 04:02:32 dist-test-slave-nhpw krb5kdc[21572](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395351, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 04:02:32.004844 21675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:31.995286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.26:45733 (local address 127.0.0.1:42663)
0926 04:02:31.995603 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:31.995611 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:31.995634 (+ 23us) server_negotiation.cc:407] Connection header received
0926 04:02:31.995753 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:31.995760 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:31.995818 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:31.995972 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:31.996762 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:31.997682 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:31.998580 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:31.999039 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:31.999176 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:32.001555 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:32.001559 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:32.001559 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:02:32.001579 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:32.003289 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:32.003683 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:32.003686 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:32.003686 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:32.003784 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:32.004017 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:32.004019 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:32.004019 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:32.004250 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:32.004425 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:32.004511 (+ 86us) server_negotiation.cc:299] Negotiation successful
0926 04:02:32.004628 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":69,"threads_started":1}
I20260926 04:02:32.005846 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 04:02:32.007874 21665 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:02:32.008538 21665 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:02:32.013979 21675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:32.007203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.26:32855 (local address 127.0.0.1:42663)
0926 04:02:32.007242 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:32.007244 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:32.007546 (+ 302us) server_negotiation.cc:407] Connection header received
0926 04:02:32.007569 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:32.007570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:32.007582 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:32.007633 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:32.008310 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:32.008927 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:32.009631 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:32.009837 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:32.009914 (+ 77us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:32.011240 (+ 1326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:32.011242 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:32.011242 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:02:32.011249 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:32.012735 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:32.013098 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:32.013099 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:32.013100 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:32.013154 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:32.013384 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:32.013385 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:32.013385 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:32.013496 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:32.013603 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:32.013733 (+ 130us) server_negotiation.cc:299] Negotiation successful
0926 04:02:32.013794 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":7,"server-negotiator.queue_time_us":20}
I20260926 04:02:32.014408 21665 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7248acc1c5cf4d7d8ccac6cd34d43c13: Generated new TSK 0
I20260926 04:02:32.014616 21665 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 04:02:32.018487 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 21585
W20260926 04:02:32.022800 21570 connection.cc:535] client connection to 127.0.0.1:42663 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42663): Connection reset by peer (error 104)
2026-09-26T04:02:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1349 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790395095474643-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 04:02:32 dist-test-slave-nhpw krb5kdc[21684](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 04:02:32 dist-test-slave-nhpw krb5kdc[21684](info): set up 2 sockets
Sep 26 04:02:32 dist-test-slave-nhpw krb5kdc[21684](info): commencing operation
krb5kdc: starting...
W20260926 04:02:33.056753 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790395095474643-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790395095474643-392-0/krb5kdc/test-user.keytab.
Sep 26 04:02:33 dist-test-slave-nhpw krb5kdc[21684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395353, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T04:02:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:02:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790395095474643-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790395095474643-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 04:02:33.176276 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790395095474643-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790395095474643-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790395095474643-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40247
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790395095474643-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40247
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:02:33.256160 21697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:02:33.256363 21697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:02:33.256385 21697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:02:33.257477 21697 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:02:33.257513 21697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:02:33.257527 21697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:02:33.257540 21697 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:02:33.257551 21697 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:02:33.258947 21697 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790395095474643-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790395095474643-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40247
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790395095474643-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40247
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21697
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790395095474643-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:02:33.259126 21697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:02:33.259289 21697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:02:33.264026 21705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:02:33.264109 21697 server_base.cc:1034] running on GCE node
W20260926 04:02:33.264073 21702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:02:33.264063 21703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:02:33.264541 21697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:02:33.264915 21697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:02:33.266112 21697 hybrid_clock.cc:648] HybridClock initialized: now 1790395353266097 us; error 48 us; skew 500 ppm
Sep 26 04:02:33 dist-test-slave-nhpw krb5kdc[21684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395353, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:02:33.268561 21697 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 04:02:33.269049 21697 webserver.cc:460] Webserver started at http://127.0.0.1:39037/ using document root <none> and password file <none>
I20260926 04:02:33.269281 21697 fs_manager.cc:359] Metadata directory not provided
I20260926 04:02:33.269326 21697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:02:33.269471 21697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:02:33.270524 21697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790395095474643-392-0/minicluster-data/master-0/data/instance:
uuid: "7dbe5300dce649d39bbd3dd5112093aa"
format_stamp: "Formatted at 2026-09-26 04:02:33 on dist-test-slave-nhpw"
server_key: "efea75201022f5771c8d9bb6e09935d4"
server_key_iv: "696a157d2365721f58836ba6dd68cff5"
server_key_version: "encryptionkey@0"
I20260926 04:02:33.270882 21697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790395095474643-392-0/minicluster-data/master-0/wal/instance:
uuid: "7dbe5300dce649d39bbd3dd5112093aa"
format_stamp: "Formatted at 2026-09-26 04:02:33 on dist-test-slave-nhpw"
server_key: "efea75201022f5771c8d9bb6e09935d4"
server_key_iv: "696a157d2365721f58836ba6dd68cff5"
server_key_version: "encryptionkey@0"
I20260926 04:02:33.272233 21697 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 04:02:33.273208 21712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:33.273491 21697 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 04:02:33.273587 21697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790395095474643-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790395095474643-392-0/minicluster-data/master-0/wal
uuid: "7dbe5300dce649d39bbd3dd5112093aa"
format_stamp: "Formatted at 2026-09-26 04:02:33 on dist-test-slave-nhpw"
server_key: "efea75201022f5771c8d9bb6e09935d4"
server_key_iv: "696a157d2365721f58836ba6dd68cff5"
server_key_version: "encryptionkey@0"
I20260926 04:02:33.273664 21697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790395095474643-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790395095474643-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790395095474643-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:02:33.285692 21697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:02:33.287593 21697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:02:33.287751 21697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:02:33.291975 21697 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40247
I20260926 04:02:33.292009 21764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40247 every 8 connection(s)
I20260926 04:02:33.292356 21697 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
I20260926 04:02:33.293358 21765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:02:33.295593 21765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dbe5300dce649d39bbd3dd5112093aa: Bootstrap starting.
I20260926 04:02:33.296176 21765 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7dbe5300dce649d39bbd3dd5112093aa: Neither blocks nor log segments found. Creating new log.
I20260926 04:02:33.296465 21765 log.cc:824] T 00000000000000000000000000000000 P 7dbe5300dce649d39bbd3dd5112093aa: Log is configured to *not* fsync() on all Append() calls
I20260926 04:02:33.297183 21765 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7dbe5300dce649d39bbd3dd5112093aa: No bootstrap required, opened a new log
I20260926 04:02:33.298751 21765 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7dbe5300dce649d39bbd3dd5112093aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dbe5300dce649d39bbd3dd5112093aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40247 } }
I20260926 04:02:33.298902 21765 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7dbe5300dce649d39bbd3dd5112093aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:02:33.298923 21765 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7dbe5300dce649d39bbd3dd5112093aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7dbe5300dce649d39bbd3dd5112093aa, State: Initialized, Role: FOLLOWER
I20260926 04:02:33.299118 21765 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7dbe5300dce649d39bbd3dd5112093aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dbe5300dce649d39bbd3dd5112093aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40247 } }
I20260926 04:02:33.299182 21765 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7dbe5300dce649d39bbd3dd5112093aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:02:33.299232 21765 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7dbe5300dce649d39bbd3dd5112093aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:02:33.299283 21765 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7dbe5300dce649d39bbd3dd5112093aa [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:02:33.299810 21765 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7dbe5300dce649d39bbd3dd5112093aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dbe5300dce649d39bbd3dd5112093aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40247 } }
I20260926 04:02:33.299893 21765 leader_election.cc:302] T 00000000000000000000000000000000 P 7dbe5300dce649d39bbd3dd5112093aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7dbe5300dce649d39bbd3dd5112093aa; no voters:
I20260926 04:02:33.300112 21765 leader_election.cc:288] T 00000000000000000000000000000000 P 7dbe5300dce649d39bbd3dd5112093aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 04:02:33.300143 21768 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7dbe5300dce649d39bbd3dd5112093aa [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:02:33.300374 21765 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7dbe5300dce649d39bbd3dd5112093aa [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:02:33.300410 21768 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7dbe5300dce649d39bbd3dd5112093aa [term 1 LEADER]: Becoming Leader. State: Replica: 7dbe5300dce649d39bbd3dd5112093aa, State: Running, Role: LEADER
I20260926 04:02:33.300581 21768 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7dbe5300dce649d39bbd3dd5112093aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dbe5300dce649d39bbd3dd5112093aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40247 } }
I20260926 04:02:33.300868 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 21697
I20260926 04:02:33.300979 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790395095474643-392-0/minicluster-data/master-0/wal/instance
I20260926 04:02:33.301190 392 external_mini_cluster.cc:1428] Setting key c5c05f0a3a08df5d36a7b19ccab31ffe
I20260926 04:02:33.301141 21769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dbe5300dce649d39bbd3dd5112093aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7dbe5300dce649d39bbd3dd5112093aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dbe5300dce649d39bbd3dd5112093aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40247 } } }
I20260926 04:02:33.301227 21769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dbe5300dce649d39bbd3dd5112093aa [sys.catalog]: This master's current role is: LEADER
I20260926 04:02:33.301550 21770 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7dbe5300dce649d39bbd3dd5112093aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7dbe5300dce649d39bbd3dd5112093aa. Latest consensus state: current_term: 1 leader_uuid: "7dbe5300dce649d39bbd3dd5112093aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7dbe5300dce649d39bbd3dd5112093aa" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40247 } } }
I20260926 04:02:33.301620 21770 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7dbe5300dce649d39bbd3dd5112093aa [sys.catalog]: This master's current role is: LEADER
I20260926 04:02:33.301764 21777 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:02:33.302286 21777 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:02:33.303795 21777 catalog_manager.cc:1347] Generated new cluster ID: dcbb3daefbb041559f57d27128f12625
I20260926 04:02:33.303848 21777 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 04:02:33.306979 21786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:33.301830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.26:55411 (local address 127.0.0.1:40247)
0926 04:02:33.302284 (+ 454us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:33.302291 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:33.302316 (+ 25us) server_negotiation.cc:407] Connection header received
0926 04:02:33.302451 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:33.302464 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:33.302510 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:33.302654 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:33.302885 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:33.304139 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:33.305043 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:33.305482 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:33.305616 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:33.306364 (+ 748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:33.306368 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:33.306369 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:02:33.306549 (+ 180us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:33.306580 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:02:33.306591 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:33.306688 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:33.306774 (+ 86us) server_negotiation.cc:299] Negotiation successful
0926 04:02:33.306804 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":66,"threads_started":1}
I20260926 04:02:33.307952 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 04:02:33.313120 21786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:33.309646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.26:44009 (local address 127.0.0.1:40247)
0926 04:02:33.309713 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:33.309715 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:33.309722 (+ 7us) server_negotiation.cc:407] Connection header received
0926 04:02:33.309739 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:33.309740 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:33.309749 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:33.309801 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:33.309976 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:33.310694 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:33.311714 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:33.312030 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:33.312096 (+ 66us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:33.312847 (+ 751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:33.312848 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:33.312848 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:02:33.312852 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:33.312868 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:02:33.312873 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:33.312939 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:33.313014 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 04:02:33.313026 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 04:02:33.313153 21777 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:02:33.313699 21777 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:02:33.316852 21777 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7dbe5300dce649d39bbd3dd5112093aa: Generated new TSK 0
I20260926 04:02:33.317026 21777 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 04:02:33.323417 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 21697
W20260926 04:02:33.327852 21682 connection.cc:535] client connection to 127.0.0.1:40247 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40247): Connection reset by peer (error 104)
2026-09-26T04:02:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1304 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790395095474643-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 04:02:33 dist-test-slave-nhpw krb5kdc[21796](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 04:02:33 dist-test-slave-nhpw krb5kdc[21796](info): set up 2 sockets
Sep 26 04:02:33 dist-test-slave-nhpw krb5kdc[21796](info): commencing operation
krb5kdc: starting...
W20260926 04:02:34.356823 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790395095474643-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790395095474643-392-0/krb5kdc/test-user.keytab.
Sep 26 04:02:34 dist-test-slave-nhpw krb5kdc[21796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395354, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T04:02:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:02:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790395095474643-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790395095474643-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 04:02:34.476799 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790395095474643-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790395095474643-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790395095474643-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42119
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790395095474643-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42119
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:02:34.556488 21809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:02:34.556753 21809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:02:34.556792 21809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:02:34.557940 21809 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:02:34.558012 21809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:02:34.558055 21809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:02:34.558096 21809 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:02:34.558120 21809 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:02:34.559566 21809 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790395095474643-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790395095474643-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42119
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790395095474643-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790395095474643-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:02:34.559840 21809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:02:34.560070 21809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:02:34.564954 21817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:02:34.564981 21815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:02:34.565016 21814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:02:34.565153 21809 server_base.cc:1034] running on GCE node
I20260926 04:02:34.565441 21809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:02:34.565693 21809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:02:34.566851 21809 hybrid_clock.cc:648] HybridClock initialized: now 1790395354566837 us; error 29 us; skew 500 ppm
Sep 26 04:02:34 dist-test-slave-nhpw krb5kdc[21796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395354, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:02:34.569375 21809 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 04:02:34.569847 21809 webserver.cc:460] Webserver started at http://127.0.0.1:40295/ using document root <none> and password file <none>
I20260926 04:02:34.570114 21809 fs_manager.cc:359] Metadata directory not provided
I20260926 04:02:34.570185 21809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:02:34.570340 21809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:02:34.571512 21809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790395095474643-392-0/minicluster-data/master-0/data/instance:
uuid: "a11c73e2f8744f719522f2fb1b60cffd"
format_stamp: "Formatted at 2026-09-26 04:02:34 on dist-test-slave-nhpw"
server_key: "9c56b378c92a95d470ff8ec302eb10a5"
server_key_iv: "7abd302d77df7b9fe0358fd14c6dbbdd"
server_key_version: "encryptionkey@0"
I20260926 04:02:34.571882 21809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790395095474643-392-0/minicluster-data/master-0/wal/instance:
uuid: "a11c73e2f8744f719522f2fb1b60cffd"
format_stamp: "Formatted at 2026-09-26 04:02:34 on dist-test-slave-nhpw"
server_key: "9c56b378c92a95d470ff8ec302eb10a5"
server_key_iv: "7abd302d77df7b9fe0358fd14c6dbbdd"
server_key_version: "encryptionkey@0"
I20260926 04:02:34.573405 21809 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 04:02:34.574241 21824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:34.574487 21809 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:02:34.574615 21809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790395095474643-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790395095474643-392-0/minicluster-data/master-0/wal
uuid: "a11c73e2f8744f719522f2fb1b60cffd"
format_stamp: "Formatted at 2026-09-26 04:02:34 on dist-test-slave-nhpw"
server_key: "9c56b378c92a95d470ff8ec302eb10a5"
server_key_iv: "7abd302d77df7b9fe0358fd14c6dbbdd"
server_key_version: "encryptionkey@0"
I20260926 04:02:34.574708 21809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790395095474643-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790395095474643-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790395095474643-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:02:34.578083 21809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:02:34.579910 21809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:02:34.580067 21809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:02:34.584256 21809 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42119
I20260926 04:02:34.584286 21876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42119 every 8 connection(s)
I20260926 04:02:34.584609 21809 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
I20260926 04:02:34.585570 21877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:02:34.587764 21877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a11c73e2f8744f719522f2fb1b60cffd: Bootstrap starting.
I20260926 04:02:34.588289 21877 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a11c73e2f8744f719522f2fb1b60cffd: Neither blocks nor log segments found. Creating new log.
I20260926 04:02:34.588521 21877 log.cc:824] T 00000000000000000000000000000000 P a11c73e2f8744f719522f2fb1b60cffd: Log is configured to *not* fsync() on all Append() calls
I20260926 04:02:34.589258 21877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a11c73e2f8744f719522f2fb1b60cffd: No bootstrap required, opened a new log
I20260926 04:02:34.590867 21877 raft_consensus.cc:348] T 00000000000000000000000000000000 P a11c73e2f8744f719522f2fb1b60cffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a11c73e2f8744f719522f2fb1b60cffd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42119 } }
I20260926 04:02:34.591005 21877 raft_consensus.cc:374] T 00000000000000000000000000000000 P a11c73e2f8744f719522f2fb1b60cffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:02:34.591029 21877 raft_consensus.cc:729] T 00000000000000000000000000000000 P a11c73e2f8744f719522f2fb1b60cffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a11c73e2f8744f719522f2fb1b60cffd, State: Initialized, Role: FOLLOWER
I20260926 04:02:34.591177 21877 consensus_queue.cc:260] T 00000000000000000000000000000000 P a11c73e2f8744f719522f2fb1b60cffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a11c73e2f8744f719522f2fb1b60cffd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42119 } }
I20260926 04:02:34.591255 21877 raft_consensus.cc:388] T 00000000000000000000000000000000 P a11c73e2f8744f719522f2fb1b60cffd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:02:34.591279 21877 raft_consensus.cc:482] T 00000000000000000000000000000000 P a11c73e2f8744f719522f2fb1b60cffd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:02:34.591307 21877 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a11c73e2f8744f719522f2fb1b60cffd [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:02:34.591532 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 21809
I20260926 04:02:34.591662 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790395095474643-392-0/minicluster-data/master-0/wal/instance
I20260926 04:02:34.591840 392 external_mini_cluster.cc:1428] Setting key b67c9952e300bffe5ad5a4e928c13a8f
I20260926 04:02:34.592043 21877 raft_consensus.cc:504] T 00000000000000000000000000000000 P a11c73e2f8744f719522f2fb1b60cffd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a11c73e2f8744f719522f2fb1b60cffd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42119 } }
I20260926 04:02:34.592128 21877 leader_election.cc:302] T 00000000000000000000000000000000 P a11c73e2f8744f719522f2fb1b60cffd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a11c73e2f8744f719522f2fb1b60cffd; no voters:
I20260926 04:02:34.592322 21877 leader_election.cc:288] T 00000000000000000000000000000000 P a11c73e2f8744f719522f2fb1b60cffd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 04:02:34.592374 21880 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a11c73e2f8744f719522f2fb1b60cffd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:02:34.592507 21880 raft_consensus.cc:686] T 00000000000000000000000000000000 P a11c73e2f8744f719522f2fb1b60cffd [term 1 LEADER]: Becoming Leader. State: Replica: a11c73e2f8744f719522f2fb1b60cffd, State: Running, Role: LEADER
I20260926 04:02:34.592571 21877 sys_catalog.cc:564] T 00000000000000000000000000000000 P a11c73e2f8744f719522f2fb1b60cffd [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:02:34.592675 21880 consensus_queue.cc:237] T 00000000000000000000000000000000 P a11c73e2f8744f719522f2fb1b60cffd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a11c73e2f8744f719522f2fb1b60cffd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42119 } }
I20260926 04:02:34.593060 21883 sys_catalog.cc:455] T 00000000000000000000000000000000 P a11c73e2f8744f719522f2fb1b60cffd [sys.catalog]: SysCatalogTable state changed. Reason: New leader a11c73e2f8744f719522f2fb1b60cffd. Latest consensus state: current_term: 1 leader_uuid: "a11c73e2f8744f719522f2fb1b60cffd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a11c73e2f8744f719522f2fb1b60cffd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42119 } } }
I20260926 04:02:34.593185 21883 sys_catalog.cc:458] T 00000000000000000000000000000000 P a11c73e2f8744f719522f2fb1b60cffd [sys.catalog]: This master's current role is: LEADER
I20260926 04:02:34.593413 21884 sys_catalog.cc:455] T 00000000000000000000000000000000 P a11c73e2f8744f719522f2fb1b60cffd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a11c73e2f8744f719522f2fb1b60cffd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a11c73e2f8744f719522f2fb1b60cffd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42119 } } }
I20260926 04:02:34.593521 21884 sys_catalog.cc:458] T 00000000000000000000000000000000 P a11c73e2f8744f719522f2fb1b60cffd [sys.catalog]: This master's current role is: LEADER
I20260926 04:02:34.593840 21886 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:02:34.594501 21886 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:02:34.594561 21882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:34.592287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.26:38557 (local address 127.0.0.1:42119)
0926 04:02:34.592447 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:34.592458 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:34.592675 (+ 217us) server_negotiation.cc:407] Connection header received
0926 04:02:34.592879 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:34.592887 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:34.592916 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:34.593065 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:34.594078 (+ 1013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:34.594082 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:34.594082 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:02:34.594086 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:34.594130 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:02:34.594144 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:34.594201 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:34.594328 (+ 127us) server_negotiation.cc:299] Negotiation successful
0926 04:02:34.594379 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":64,"threads_started":1}
I20260926 04:02:34.595515 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 04:02:34.597396 21886 catalog_manager.cc:1347] Generated new cluster ID: f1615ee39b394d50bfa0a1d5a8e7c96a
I20260926 04:02:34.597456 21886 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 04:02:34.598448 21882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:34.597068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.26:44267 (local address 127.0.0.1:42119)
0926 04:02:34.597106 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:34.597108 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:34.597488 (+ 380us) server_negotiation.cc:407] Connection header received
0926 04:02:34.597584 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:34.597585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:34.597591 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:34.597643 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:34.598228 (+ 585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:34.598229 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:34.598230 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:02:34.598233 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:34.598253 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:02:34.598257 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:34.598308 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:34.598376 (+ 68us) server_negotiation.cc:299] Negotiation successful
0926 04:02:34.598388 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 04:02:34.613370 21886 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:02:34.613548 21886 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:02:34.617369 21886 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a11c73e2f8744f719522f2fb1b60cffd: Generated new TSK 0
I20260926 04:02:34.617542 21886 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 04:02:34.618976 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 21809
2026-09-26T04:02:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1297 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790395095474643-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 04:02:34 dist-test-slave-nhpw krb5kdc[21908](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 04:02:34 dist-test-slave-nhpw krb5kdc[21908](info): set up 2 sockets
Sep 26 04:02:34 dist-test-slave-nhpw krb5kdc[21908](info): commencing operation
krb5kdc: starting...
W20260926 04:02:35.656847 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790395095474643-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790395095474643-392-0/krb5kdc/test-user.keytab.
Sep 26 04:02:35 dist-test-slave-nhpw krb5kdc[21908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T04:02:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:02:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 04:02:35.771694 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790395095474643-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790395095474643-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790395095474643-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39843
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35551
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39843
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:02:35.850471 21921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:02:35.850704 21921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:02:35.850740 21921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:02:35.851874 21921 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:02:35.851948 21921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:02:35.851974 21921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:02:35.852001 21921 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:02:35.852032 21921 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:02:35.853525 21921 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35551
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790395095474643-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790395095474643-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39843
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21921
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790395095474643-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:02:35.853792 21921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:02:35.854017 21921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:02:35.858735 21926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:02:35.858759 21927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:02:35.858760 21929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:02:35.858929 21921 server_base.cc:1034] running on GCE node
I20260926 04:02:35.859138 21921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:02:35.859361 21921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:02:35.860524 21921 hybrid_clock.cc:648] HybridClock initialized: now 1790395355860511 us; error 28 us; skew 500 ppm
Sep 26 04:02:35 dist-test-slave-nhpw krb5kdc[21908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395355, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:02:35.863000 21921 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 04:02:35.863420 21921 webserver.cc:460] Webserver started at http://127.0.98.62:41127/ using document root <none> and password file <none>
I20260926 04:02:35.863648 21921 fs_manager.cc:359] Metadata directory not provided
I20260926 04:02:35.863694 21921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:02:35.863863 21921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:02:35.864964 21921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790395095474643-392-0/minicluster-data/master-0/data/instance:
uuid: "31f8fc522b714153823a123925f19c4b"
format_stamp: "Formatted at 2026-09-26 04:02:35 on dist-test-slave-nhpw"
server_key: "b2e2a19e21317d29896b2a57499d14a1"
server_key_iv: "75cbda31b8f1b63753e092915a1766a9"
server_key_version: "encryptionkey@0"
I20260926 04:02:35.865335 21921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790395095474643-392-0/minicluster-data/master-0/wal/instance:
uuid: "31f8fc522b714153823a123925f19c4b"
format_stamp: "Formatted at 2026-09-26 04:02:35 on dist-test-slave-nhpw"
server_key: "b2e2a19e21317d29896b2a57499d14a1"
server_key_iv: "75cbda31b8f1b63753e092915a1766a9"
server_key_version: "encryptionkey@0"
I20260926 04:02:35.866665 21921 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 04:02:35.867462 21936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:35.867641 21921 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 04:02:35.867759 21921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790395095474643-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790395095474643-392-0/minicluster-data/master-0/wal
uuid: "31f8fc522b714153823a123925f19c4b"
format_stamp: "Formatted at 2026-09-26 04:02:35 on dist-test-slave-nhpw"
server_key: "b2e2a19e21317d29896b2a57499d14a1"
server_key_iv: "75cbda31b8f1b63753e092915a1766a9"
server_key_version: "encryptionkey@0"
I20260926 04:02:35.867834 21921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790395095474643-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790395095474643-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790395095474643-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:02:35.894496 21921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:02:35.896502 21921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:02:35.896731 21921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:02:35.900893 21921 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39843
I20260926 04:02:35.900949 21988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39843 every 8 connection(s)
I20260926 04:02:35.901314 21921 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
I20260926 04:02:35.902139 21989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:02:35.904284 21989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31f8fc522b714153823a123925f19c4b: Bootstrap starting.
I20260926 04:02:35.904827 21989 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31f8fc522b714153823a123925f19c4b: Neither blocks nor log segments found. Creating new log.
I20260926 04:02:35.905061 21989 log.cc:824] T 00000000000000000000000000000000 P 31f8fc522b714153823a123925f19c4b: Log is configured to *not* fsync() on all Append() calls
I20260926 04:02:35.905709 21989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31f8fc522b714153823a123925f19c4b: No bootstrap required, opened a new log
I20260926 04:02:35.906523 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 21921
I20260926 04:02:35.906675 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790395095474643-392-0/minicluster-data/master-0/wal/instance
I20260926 04:02:35.906863 392 external_mini_cluster.cc:1428] Setting key 98c88bb40b1b5703a341007d63b73e8b
I20260926 04:02:35.907691 21989 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31f8fc522b714153823a123925f19c4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31f8fc522b714153823a123925f19c4b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39843 } }
I20260926 04:02:35.907859 21989 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31f8fc522b714153823a123925f19c4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:02:35.907899 21989 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31f8fc522b714153823a123925f19c4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31f8fc522b714153823a123925f19c4b, State: Initialized, Role: FOLLOWER
I20260926 04:02:35.908032 21989 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31f8fc522b714153823a123925f19c4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31f8fc522b714153823a123925f19c4b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39843 } }
I20260926 04:02:35.908119 21989 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31f8fc522b714153823a123925f19c4b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:02:35.908182 21989 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31f8fc522b714153823a123925f19c4b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:02:35.908229 21989 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31f8fc522b714153823a123925f19c4b [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:02:35.909310 21989 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31f8fc522b714153823a123925f19c4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31f8fc522b714153823a123925f19c4b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39843 } }
I20260926 04:02:35.909420 21989 leader_election.cc:302] T 00000000000000000000000000000000 P 31f8fc522b714153823a123925f19c4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31f8fc522b714153823a123925f19c4b; no voters:
I20260926 04:02:35.909562 21989 leader_election.cc:288] T 00000000000000000000000000000000 P 31f8fc522b714153823a123925f19c4b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 04:02:35.909626 21994 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31f8fc522b714153823a123925f19c4b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:02:35.909801 21994 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31f8fc522b714153823a123925f19c4b [term 1 LEADER]: Becoming Leader. State: Replica: 31f8fc522b714153823a123925f19c4b, State: Running, Role: LEADER
I20260926 04:02:35.909816 21989 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31f8fc522b714153823a123925f19c4b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:02:35.909902 21994 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31f8fc522b714153823a123925f19c4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31f8fc522b714153823a123925f19c4b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39843 } }
I20260926 04:02:35.910288 21994 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31f8fc522b714153823a123925f19c4b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31f8fc522b714153823a123925f19c4b. Latest consensus state: current_term: 1 leader_uuid: "31f8fc522b714153823a123925f19c4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31f8fc522b714153823a123925f19c4b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39843 } } }
I20260926 04:02:35.910408 21994 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31f8fc522b714153823a123925f19c4b [sys.catalog]: This master's current role is: LEADER
I20260926 04:02:35.910696 22003 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:02:35.911273 22003 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:02:35.910300 21995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31f8fc522b714153823a123925f19c4b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31f8fc522b714153823a123925f19c4b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31f8fc522b714153823a123925f19c4b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39843 } } }
I20260926 04:02:35.912184 21995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31f8fc522b714153823a123925f19c4b [sys.catalog]: This master's current role is: LEADER
I20260926 04:02:35.913175 22003 catalog_manager.cc:1347] Generated new cluster ID: ee902099697a495aafd5e942336904f7
I20260926 04:02:35.913228 22003 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 04:02:35 dist-test-slave-nhpw krb5kdc[21908](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 04:02:35.917781 21993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:35.907376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44227 (local address 127.0.98.62:39843)
0926 04:02:35.907580 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:35.907590 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:35.907626 (+ 36us) server_negotiation.cc:407] Connection header received
0926 04:02:35.907737 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:35.907744 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:35.907798 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:35.907967 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:35.908788 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:35.910265 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:35.911494 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:35.911971 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:35.912050 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:35.914528 (+ 2478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:35.914531 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:35.914532 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:02:35.914543 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:35.916194 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:35.916584 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:35.916587 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:35.916588 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:35.916749 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:35.916977 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:35.916978 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:35.916978 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:35.917209 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:35.917400 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:35.917483 (+ 83us) server_negotiation.cc:299] Negotiation successful
0926 04:02:35.917585 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":102,"threads_started":1}
I20260926 04:02:35.918751 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 04:02:35.926216 21993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:35.920056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42749 (local address 127.0.98.62:39843)
0926 04:02:35.920093 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:35.920095 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:35.920216 (+ 121us) server_negotiation.cc:407] Connection header received
0926 04:02:35.920287 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:35.920288 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:35.920297 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:35.920352 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:35.920909 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:35.921395 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:35.922096 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:35.922320 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:35.922412 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:35.923555 (+ 1143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:35.923556 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:35.923557 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:02:35.923563 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:35.925041 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:35.925403 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:35.925404 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:35.925405 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:35.925456 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:35.925685 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:35.925685 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:35.925686 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:35.925793 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:35.925907 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:35.926035 (+ 128us) server_negotiation.cc:299] Negotiation successful
0926 04:02:35.926104 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 04:02:35.931257 22003 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:02:35.931766 22003 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:02:35.941262 22003 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 31f8fc522b714153823a123925f19c4b: Generated new TSK 0
I20260926 04:02:35.941425 22003 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 04:02:35.944864 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 21921
W20260926 04:02:35.949154 21906 connection.cc:535] client connection to 127.0.98.62:39843 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:39843): Connection reset by peer (error 104)
2026-09-26T04:02:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1324 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790395095474643-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 04:02:36 dist-test-slave-nhpw krb5kdc[22020](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 04:02:36 dist-test-slave-nhpw krb5kdc[22020](info): set up 2 sockets
Sep 26 04:02:36 dist-test-slave-nhpw krb5kdc[22020](info): commencing operation
krb5kdc: starting...
W20260926 04:02:36.992589 392 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790395095474643-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790395095474643-392-0/krb5kdc/test-user.keytab.
Sep 26 04:02:37 dist-test-slave-nhpw krb5kdc[22020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395357, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T04:02:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:02:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 04:02:37.108161 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790395095474643-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790395095474643-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790395095474643-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40069
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40069
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:02:37.186663 22033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:02:37.186867 22033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:02:37.186889 22033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:02:37.188035 22033 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:02:37.188078 22033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:02:37.188092 22033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:02:37.188105 22033 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:02:37.188165 22033 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:02:37.189688 22033 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790395095474643-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790395095474643-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40069
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790395095474643-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:02:37.189934 22033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:02:37.190157 22033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:02:37.194906 22041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:02:37.194938 22039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:02:37.195040 22033 server_base.cc:1034] running on GCE node
W20260926 04:02:37.195139 22038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:02:37.195389 22033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:02:37.195679 22033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:02:37.196851 22033 hybrid_clock.cc:648] HybridClock initialized: now 1790395357196827 us; error 39 us; skew 500 ppm
Sep 26 04:02:37 dist-test-slave-nhpw krb5kdc[22020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395357, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:02:37.203788 22033 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 04:02:37.204229 22033 webserver.cc:460] Webserver started at http://127.0.98.62:42391/ using document root <none> and password file <none>
I20260926 04:02:37.204519 22033 fs_manager.cc:359] Metadata directory not provided
I20260926 04:02:37.204602 22033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:02:37.204753 22033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:02:37.205829 22033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790395095474643-392-0/minicluster-data/master-0/data/instance:
uuid: "2f9065e5a30040f8a9977993f2d0a1bb"
format_stamp: "Formatted at 2026-09-26 04:02:37 on dist-test-slave-nhpw"
server_key: "b81820ea08e607ca720157fe8aac97cf"
server_key_iv: "00e780e719656b858ab1a7b60e5603cb"
server_key_version: "encryptionkey@0"
I20260926 04:02:37.206211 22033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790395095474643-392-0/minicluster-data/master-0/wal/instance:
uuid: "2f9065e5a30040f8a9977993f2d0a1bb"
format_stamp: "Formatted at 2026-09-26 04:02:37 on dist-test-slave-nhpw"
server_key: "b81820ea08e607ca720157fe8aac97cf"
server_key_iv: "00e780e719656b858ab1a7b60e5603cb"
server_key_version: "encryptionkey@0"
I20260926 04:02:37.207557 22033 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 04:02:37.208387 22048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:37.208583 22033 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 04:02:37.208703 22033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790395095474643-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790395095474643-392-0/minicluster-data/master-0/wal
uuid: "2f9065e5a30040f8a9977993f2d0a1bb"
format_stamp: "Formatted at 2026-09-26 04:02:37 on dist-test-slave-nhpw"
server_key: "b81820ea08e607ca720157fe8aac97cf"
server_key_iv: "00e780e719656b858ab1a7b60e5603cb"
server_key_version: "encryptionkey@0"
I20260926 04:02:37.208791 22033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790395095474643-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790395095474643-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790395095474643-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:02:37.234063 22033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:02:37.235986 22033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:02:37.236155 22033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:02:37.240144 22033 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40069
I20260926 04:02:37.240190 22100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40069 every 8 connection(s)
I20260926 04:02:37.240587 22033 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
I20260926 04:02:37.241442 22101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:02:37.242652 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 22033
I20260926 04:02:37.242784 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790395095474643-392-0/minicluster-data/master-0/wal/instance
I20260926 04:02:37.242951 392 external_mini_cluster.cc:1428] Setting key 92320ac022cc2de0582b7dd4a086bde5
I20260926 04:02:37.243959 22101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f9065e5a30040f8a9977993f2d0a1bb: Bootstrap starting.
I20260926 04:02:37.244715 22101 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f9065e5a30040f8a9977993f2d0a1bb: Neither blocks nor log segments found. Creating new log.
I20260926 04:02:37.244997 22101 log.cc:824] T 00000000000000000000000000000000 P 2f9065e5a30040f8a9977993f2d0a1bb: Log is configured to *not* fsync() on all Append() calls
I20260926 04:02:37.245738 22101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f9065e5a30040f8a9977993f2d0a1bb: No bootstrap required, opened a new log
I20260926 04:02:37.248026 22101 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f9065e5a30040f8a9977993f2d0a1bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f9065e5a30040f8a9977993f2d0a1bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40069 } }
I20260926 04:02:37.248199 22101 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f9065e5a30040f8a9977993f2d0a1bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:02:37.248246 22101 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f9065e5a30040f8a9977993f2d0a1bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f9065e5a30040f8a9977993f2d0a1bb, State: Initialized, Role: FOLLOWER
I20260926 04:02:37.248414 22101 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f9065e5a30040f8a9977993f2d0a1bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f9065e5a30040f8a9977993f2d0a1bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40069 } }
I20260926 04:02:37.248504 22101 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f9065e5a30040f8a9977993f2d0a1bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:02:37.248528 22101 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f9065e5a30040f8a9977993f2d0a1bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:02:37.248593 22101 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f9065e5a30040f8a9977993f2d0a1bb [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:02:37.248595 22104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:37.243458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49383 (local address 127.0.98.62:40069)
0926 04:02:37.243641 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:37.243648 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:37.243672 (+ 24us) server_negotiation.cc:407] Connection header received
0926 04:02:37.244083 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:37.244092 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:37.244149 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:37.244280 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:37.244528 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:37.245549 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:37.246656 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:37.247056 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:37.247138 (+ 82us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:37.247851 (+ 713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:37.247855 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:37.247855 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:02:37.248013 (+ 158us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:37.248042 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:02:37.248052 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:37.248116 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:37.248280 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 04:02:37.248310 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":101,"threads_started":1}
I20260926 04:02:37.249308 22101 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f9065e5a30040f8a9977993f2d0a1bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f9065e5a30040f8a9977993f2d0a1bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40069 } }
I20260926 04:02:37.249418 22101 leader_election.cc:302] T 00000000000000000000000000000000 P 2f9065e5a30040f8a9977993f2d0a1bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f9065e5a30040f8a9977993f2d0a1bb; no voters:
I20260926 04:02:37.249577 22101 leader_election.cc:288] T 00000000000000000000000000000000 P 2f9065e5a30040f8a9977993f2d0a1bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 04:02:37.249590 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 04:02:37.249607 22106 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f9065e5a30040f8a9977993f2d0a1bb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:02:37.249820 22101 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f9065e5a30040f8a9977993f2d0a1bb [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:02:37.249830 22106 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f9065e5a30040f8a9977993f2d0a1bb [term 1 LEADER]: Becoming Leader. State: Replica: 2f9065e5a30040f8a9977993f2d0a1bb, State: Running, Role: LEADER
I20260926 04:02:37.250054 22106 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f9065e5a30040f8a9977993f2d0a1bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f9065e5a30040f8a9977993f2d0a1bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40069 } }
I20260926 04:02:37.250363 22106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f9065e5a30040f8a9977993f2d0a1bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f9065e5a30040f8a9977993f2d0a1bb. Latest consensus state: current_term: 1 leader_uuid: "2f9065e5a30040f8a9977993f2d0a1bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f9065e5a30040f8a9977993f2d0a1bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40069 } } }
I20260926 04:02:37.250376 22108 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f9065e5a30040f8a9977993f2d0a1bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f9065e5a30040f8a9977993f2d0a1bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f9065e5a30040f8a9977993f2d0a1bb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40069 } } }
I20260926 04:02:37.250504 22108 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f9065e5a30040f8a9977993f2d0a1bb [sys.catalog]: This master's current role is: LEADER
I20260926 04:02:37.250679 22106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f9065e5a30040f8a9977993f2d0a1bb [sys.catalog]: This master's current role is: LEADER
I20260926 04:02:37.251159 22113 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:02:37.251644 22113 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:02:37.253938 22113 catalog_manager.cc:1347] Generated new cluster ID: 18a4a4a866604dd0a2dd46de0b5af096
I20260926 04:02:37.253990 22113 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 04:02:37.254957 22104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:37.251119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42545 (local address 127.0.98.62:40069)
0926 04:02:37.251151 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:37.251153 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:37.251725 (+ 572us) server_negotiation.cc:407] Connection header received
0926 04:02:37.251892 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:37.251893 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:37.251905 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:37.251976 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:37.252124 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:37.252872 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:37.253625 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:37.253926 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:37.254001 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:37.254682 (+ 681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:37.254683 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:37.254684 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:02:37.254688 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:37.254705 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:02:37.254709 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:37.254778 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:37.254873 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 04:02:37.254885 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 04:02:37.269783 22113 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:02:37.270336 22113 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:02:37.278879 22113 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f9065e5a30040f8a9977993f2d0a1bb: Generated new TSK 0
I20260926 04:02:37.279127 22113 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 04:02:37.282164 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 22033
W20260926 04:02:37.286579 22018 connection.cc:535] client connection to 127.0.98.62:40069 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:40069): Connection reset by peer (error 104)
2026-09-26T04:02:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1337 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790395095474643-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 04:02:37 dist-test-slave-nhpw krb5kdc[22132](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 04:02:37 dist-test-slave-nhpw krb5kdc[22132](info): set up 2 sockets
Sep 26 04:02:37 dist-test-slave-nhpw krb5kdc[22132](info): commencing operation
krb5kdc: starting...
W20260926 04:02:38.312855 392 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790395095474643-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790395095474643-392-0/krb5kdc/test-user.keytab.
Sep 26 04:02:38 dist-test-slave-nhpw krb5kdc[22132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395358, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T04:02:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:02:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 04:02:38.431450 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790395095474643-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790395095474643-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790395095474643-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40727
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40727
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:02:38.516256 22145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:02:38.516443 22145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:02:38.516463 22145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:02:38.517581 22145 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:02:38.517629 22145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:02:38.517643 22145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:02:38.517656 22145 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:02:38.517668 22145 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:02:38.519088 22145 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790395095474643-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790395095474643-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40727
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790395095474643-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:02:38.519332 22145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:02:38.519570 22145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:02:38.524312 22153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:02:38.524397 22145 server_base.cc:1034] running on GCE node
W20260926 04:02:38.524279 22151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:02:38.524276 22150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:02:38.524765 22145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:02:38.524992 22145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:02:38.526137 22145 hybrid_clock.cc:648] HybridClock initialized: now 1790395358526120 us; error 27 us; skew 500 ppm
Sep 26 04:02:38 dist-test-slave-nhpw krb5kdc[22132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395358, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:02:38.528576 22145 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 04:02:38.529007 22145 webserver.cc:460] Webserver started at http://127.0.98.62:39783/ using document root <none> and password file <none>
I20260926 04:02:38.529228 22145 fs_manager.cc:359] Metadata directory not provided
I20260926 04:02:38.529296 22145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:02:38.529449 22145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:02:38.530478 22145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790395095474643-392-0/minicluster-data/master-0/data/instance:
uuid: "cd7810a92dbc48e19f857339b461c3cc"
format_stamp: "Formatted at 2026-09-26 04:02:38 on dist-test-slave-nhpw"
server_key: "21394714b35495177687d89315a97e12"
server_key_iv: "fede9bfaec00ed9999eed0308f8620ca"
server_key_version: "encryptionkey@0"
I20260926 04:02:38.530843 22145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790395095474643-392-0/minicluster-data/master-0/wal/instance:
uuid: "cd7810a92dbc48e19f857339b461c3cc"
format_stamp: "Formatted at 2026-09-26 04:02:38 on dist-test-slave-nhpw"
server_key: "21394714b35495177687d89315a97e12"
server_key_iv: "fede9bfaec00ed9999eed0308f8620ca"
server_key_version: "encryptionkey@0"
I20260926 04:02:38.532295 22145 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 04:02:38.533200 22160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:38.533406 22145 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 04:02:38.533512 22145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790395095474643-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790395095474643-392-0/minicluster-data/master-0/wal
uuid: "cd7810a92dbc48e19f857339b461c3cc"
format_stamp: "Formatted at 2026-09-26 04:02:38 on dist-test-slave-nhpw"
server_key: "21394714b35495177687d89315a97e12"
server_key_iv: "fede9bfaec00ed9999eed0308f8620ca"
server_key_version: "encryptionkey@0"
I20260926 04:02:38.533587 22145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790395095474643-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790395095474643-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790395095474643-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:02:38.536913 22145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:02:38.539008 22145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:02:38.539108 22145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:02:38.542781 22145 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40727
I20260926 04:02:38.542812 22212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40727 every 8 connection(s)
I20260926 04:02:38.543118 22145 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
I20260926 04:02:38.544064 22213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:02:38.546331 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 22145
I20260926 04:02:38.546334 22213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd7810a92dbc48e19f857339b461c3cc: Bootstrap starting.
I20260926 04:02:38.546461 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790395095474643-392-0/minicluster-data/master-0/wal/instance
I20260926 04:02:38.546654 392 external_mini_cluster.cc:1428] Setting key 0b136d3e997ebf3d5cadf2b93f835438
I20260926 04:02:38.547013 22213 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd7810a92dbc48e19f857339b461c3cc: Neither blocks nor log segments found. Creating new log.
I20260926 04:02:38.547375 22213 log.cc:824] T 00000000000000000000000000000000 P cd7810a92dbc48e19f857339b461c3cc: Log is configured to *not* fsync() on all Append() calls
I20260926 04:02:38.548094 22213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd7810a92dbc48e19f857339b461c3cc: No bootstrap required, opened a new log
I20260926 04:02:38.549209 22216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:38.547107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38885 (local address 127.0.98.62:40727)
0926 04:02:38.547302 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:38.547314 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:38.547338 (+ 24us) server_negotiation.cc:407] Connection header received
0926 04:02:38.547952 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:38.547959 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:38.547986 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:38.548115 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:38.548771 (+ 656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:38.548774 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:38.548775 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:02:38.548778 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:38.548812 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:02:38.548822 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:38.548888 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:38.548956 (+ 68us) server_negotiation.cc:299] Negotiation successful
0926 04:02:38.549009 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":93,"threads_started":1}
I20260926 04:02:38.550179 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 04:02:38.550627 22213 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd7810a92dbc48e19f857339b461c3cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd7810a92dbc48e19f857339b461c3cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40727 } }
I20260926 04:02:38.550801 22213 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd7810a92dbc48e19f857339b461c3cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:02:38.550834 22213 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd7810a92dbc48e19f857339b461c3cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd7810a92dbc48e19f857339b461c3cc, State: Initialized, Role: FOLLOWER
I20260926 04:02:38.550949 22213 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd7810a92dbc48e19f857339b461c3cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd7810a92dbc48e19f857339b461c3cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40727 } }
I20260926 04:02:38.551046 22213 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd7810a92dbc48e19f857339b461c3cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:02:38.551079 22213 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd7810a92dbc48e19f857339b461c3cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:02:38.551124 22213 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd7810a92dbc48e19f857339b461c3cc [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:02:38.551843 22213 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd7810a92dbc48e19f857339b461c3cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd7810a92dbc48e19f857339b461c3cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40727 } }
I20260926 04:02:38.551955 22213 leader_election.cc:302] T 00000000000000000000000000000000 P cd7810a92dbc48e19f857339b461c3cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd7810a92dbc48e19f857339b461c3cc; no voters:
I20260926 04:02:38.552112 22213 leader_election.cc:288] T 00000000000000000000000000000000 P cd7810a92dbc48e19f857339b461c3cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 04:02:38.552134 22223 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd7810a92dbc48e19f857339b461c3cc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:02:38.552264 22223 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd7810a92dbc48e19f857339b461c3cc [term 1 LEADER]: Becoming Leader. State: Replica: cd7810a92dbc48e19f857339b461c3cc, State: Running, Role: LEADER
I20260926 04:02:38.552364 22213 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd7810a92dbc48e19f857339b461c3cc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:02:38.552380 22223 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd7810a92dbc48e19f857339b461c3cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd7810a92dbc48e19f857339b461c3cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40727 } }
I20260926 04:02:38.552702 22223 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd7810a92dbc48e19f857339b461c3cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd7810a92dbc48e19f857339b461c3cc. Latest consensus state: current_term: 1 leader_uuid: "cd7810a92dbc48e19f857339b461c3cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd7810a92dbc48e19f857339b461c3cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40727 } } }
I20260926 04:02:38.552871 22223 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd7810a92dbc48e19f857339b461c3cc [sys.catalog]: This master's current role is: LEADER
I20260926 04:02:38.552714 22224 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd7810a92dbc48e19f857339b461c3cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd7810a92dbc48e19f857339b461c3cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd7810a92dbc48e19f857339b461c3cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40727 } } }
I20260926 04:02:38.552974 22216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:38.551771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57731 (local address 127.0.98.62:40727)
0926 04:02:38.551824 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:38.551826 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:38.551940 (+ 114us) server_negotiation.cc:407] Connection header received
0926 04:02:38.551999 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:38.552000 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:38.552004 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:38.552058 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:38.552686 (+ 628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:38.552687 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:38.552687 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:02:38.552691 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:38.552705 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:02:38.552709 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:38.552755 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:38.552892 (+ 137us) server_negotiation.cc:299] Negotiation successful
0926 04:02:38.552905 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 04:02:38.553061 22224 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd7810a92dbc48e19f857339b461c3cc [sys.catalog]: This master's current role is: LEADER
I20260926 04:02:38.553483 22232 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:02:38.553996 22232 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:02:38.555546 22232 catalog_manager.cc:1347] Generated new cluster ID: e8792158ccb14a179a2efcd653119cce
I20260926 04:02:38.555584 22232 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 04:02:38.567528 22232 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:02:38.567685 22232 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:02:38.569439 22232 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd7810a92dbc48e19f857339b461c3cc: Generated new TSK 0
I20260926 04:02:38.569602 22232 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 04:02:38.576196 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 22145
2026-09-26T04:02:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1293 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790395095474643-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 04:02:38 dist-test-slave-nhpw krb5kdc[22244](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 04:02:38 dist-test-slave-nhpw krb5kdc[22244](info): set up 2 sockets
Sep 26 04:02:38 dist-test-slave-nhpw krb5kdc[22244](info): commencing operation
krb5kdc: starting...
W20260926 04:02:39.608944 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790395095474643-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790395095474643-392-0/krb5kdc/test-user.keytab.
Sep 26 04:02:39 dist-test-slave-nhpw krb5kdc[22244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T04:02:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:02:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 04:02:39.728360 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790395095474643-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790395095474643-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790395095474643-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41603
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41603
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:02:39.807420 22257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:02:39.807641 22257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:02:39.807664 22257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:02:39.808825 22257 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:02:39.808866 22257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:02:39.808879 22257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:02:39.808933 22257 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:02:39.808950 22257 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:02:39.810374 22257 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790395095474643-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790395095474643-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41603
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790395095474643-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:02:39.810616 22257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:02:39.810853 22257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 04:02:39.815610 22257 server_base.cc:1034] running on GCE node
W20260926 04:02:39.815590 22262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:02:39.815570 22263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:02:39.815560 22265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:02:39.816076 22257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:02:39.816406 22257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:02:39.817567 22257 hybrid_clock.cc:648] HybridClock initialized: now 1790395359817556 us; error 45 us; skew 500 ppm
Sep 26 04:02:39 dist-test-slave-nhpw krb5kdc[22244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395359, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:02:39.820111 22257 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 04:02:39.820519 22257 webserver.cc:460] Webserver started at http://127.0.98.62:36243/ using document root <none> and password file <none>
I20260926 04:02:39.820780 22257 fs_manager.cc:359] Metadata directory not provided
I20260926 04:02:39.820827 22257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:02:39.820915 22257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:02:39.821873 22257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790395095474643-392-0/minicluster-data/master-0/data/instance:
uuid: "bb40278afbbf49d086fcebe726d0779d"
format_stamp: "Formatted at 2026-09-26 04:02:39 on dist-test-slave-nhpw"
server_key: "a0a35fd6df40e2a69897fddade206d3b"
server_key_iv: "d4584bcd2ad25a0dd5558d324ac24963"
server_key_version: "encryptionkey@0"
I20260926 04:02:39.822229 22257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790395095474643-392-0/minicluster-data/master-0/wal/instance:
uuid: "bb40278afbbf49d086fcebe726d0779d"
format_stamp: "Formatted at 2026-09-26 04:02:39 on dist-test-slave-nhpw"
server_key: "a0a35fd6df40e2a69897fddade206d3b"
server_key_iv: "d4584bcd2ad25a0dd5558d324ac24963"
server_key_version: "encryptionkey@0"
I20260926 04:02:39.823462 22257 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 04:02:39.824134 22272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:39.824301 22257 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 04:02:39.824390 22257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790395095474643-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790395095474643-392-0/minicluster-data/master-0/wal
uuid: "bb40278afbbf49d086fcebe726d0779d"
format_stamp: "Formatted at 2026-09-26 04:02:39 on dist-test-slave-nhpw"
server_key: "a0a35fd6df40e2a69897fddade206d3b"
server_key_iv: "d4584bcd2ad25a0dd5558d324ac24963"
server_key_version: "encryptionkey@0"
I20260926 04:02:39.824450 22257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790395095474643-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790395095474643-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790395095474643-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:02:39.840591 22257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:02:39.842540 22257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:02:39.842664 22257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:02:39.846498 22257 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41603
I20260926 04:02:39.846551 22324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41603 every 8 connection(s)
I20260926 04:02:39.846908 22257 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
I20260926 04:02:39.847792 22325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:02:39.850057 22325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb40278afbbf49d086fcebe726d0779d: Bootstrap starting.
I20260926 04:02:39.850636 22325 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb40278afbbf49d086fcebe726d0779d: Neither blocks nor log segments found. Creating new log.
I20260926 04:02:39.850939 22325 log.cc:824] T 00000000000000000000000000000000 P bb40278afbbf49d086fcebe726d0779d: Log is configured to *not* fsync() on all Append() calls
I20260926 04:02:39.851637 22325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb40278afbbf49d086fcebe726d0779d: No bootstrap required, opened a new log
I20260926 04:02:39.852521 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 22257
I20260926 04:02:39.852614 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790395095474643-392-0/minicluster-data/master-0/wal/instance
I20260926 04:02:39.852838 392 external_mini_cluster.cc:1428] Setting key 8a8975fcf56ac88cb2bdd7f0f40a4711
I20260926 04:02:39.853886 22325 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb40278afbbf49d086fcebe726d0779d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb40278afbbf49d086fcebe726d0779d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41603 } }
I20260926 04:02:39.854065 22325 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb40278afbbf49d086fcebe726d0779d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:02:39.854104 22325 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb40278afbbf49d086fcebe726d0779d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb40278afbbf49d086fcebe726d0779d, State: Initialized, Role: FOLLOWER
I20260926 04:02:39.854269 22325 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb40278afbbf49d086fcebe726d0779d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb40278afbbf49d086fcebe726d0779d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41603 } }
I20260926 04:02:39.854360 22325 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb40278afbbf49d086fcebe726d0779d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:02:39.854398 22325 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb40278afbbf49d086fcebe726d0779d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:02:39.854449 22325 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb40278afbbf49d086fcebe726d0779d [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:02:39.855449 22325 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb40278afbbf49d086fcebe726d0779d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb40278afbbf49d086fcebe726d0779d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41603 } }
I20260926 04:02:39.855538 22325 leader_election.cc:302] T 00000000000000000000000000000000 P bb40278afbbf49d086fcebe726d0779d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb40278afbbf49d086fcebe726d0779d; no voters:
I20260926 04:02:39.855718 22325 leader_election.cc:288] T 00000000000000000000000000000000 P bb40278afbbf49d086fcebe726d0779d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 04:02:39.855732 22330 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb40278afbbf49d086fcebe726d0779d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:02:39.855859 22330 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb40278afbbf49d086fcebe726d0779d [term 1 LEADER]: Becoming Leader. State: Replica: bb40278afbbf49d086fcebe726d0779d, State: Running, Role: LEADER
I20260926 04:02:39.855990 22325 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb40278afbbf49d086fcebe726d0779d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:02:39.855993 22330 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb40278afbbf49d086fcebe726d0779d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb40278afbbf49d086fcebe726d0779d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41603 } }
I20260926 04:02:39.857074 22330 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb40278afbbf49d086fcebe726d0779d [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb40278afbbf49d086fcebe726d0779d. Latest consensus state: current_term: 1 leader_uuid: "bb40278afbbf49d086fcebe726d0779d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb40278afbbf49d086fcebe726d0779d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41603 } } }
I20260926 04:02:39.857182 22330 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb40278afbbf49d086fcebe726d0779d [sys.catalog]: This master's current role is: LEADER
I20260926 04:02:39.857374 22331 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb40278afbbf49d086fcebe726d0779d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb40278afbbf49d086fcebe726d0779d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb40278afbbf49d086fcebe726d0779d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41603 } } }
I20260926 04:02:39.858449 22331 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb40278afbbf49d086fcebe726d0779d [sys.catalog]: This master's current role is: LEADER
W20260926 04:02:39.858510 22344 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bb40278afbbf49d086fcebe726d0779d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 04:02:39.858616 22344 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 04:02:39.859299 22345 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:02:39.859488 22345 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 04:02:39 dist-test-slave-nhpw krb5kdc[22244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395359, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 04:02:39.861873 22345 catalog_manager.cc:1347] Generated new cluster ID: ea1faee01e2743d29af9ee7597c1a3e6
I20260926 04:02:39.861943 22345 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 04:02:39.864423 22329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:39.853492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45085 (local address 127.0.98.62:41603)
0926 04:02:39.853764 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:39.853774 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:39.853805 (+ 31us) server_negotiation.cc:407] Connection header received
0926 04:02:39.853911 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:39.853916 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:39.853957 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:39.854089 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:39.854899 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:39.855971 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:39.857173 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:39.857712 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:39.857797 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:39.860754 (+ 2957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:39.860760 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:39.860761 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:02:39.860775 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:39.862796 (+ 2021us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:39.863184 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:39.863188 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:39.863188 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:39.863321 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:39.863524 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:39.863525 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:39.863525 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:39.863830 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:39.863988 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:39.864061 (+ 73us) server_negotiation.cc:299] Negotiation successful
0926 04:02:39.864210 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":110,"threads_started":1}
I20260926 04:02:39.865670 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 04:02:39.867295 22345 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:02:39.867880 22345 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:02:39.874598 22329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:39.867168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34315 (local address 127.0.98.62:41603)
0926 04:02:39.867296 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:39.867298 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:39.867455 (+ 157us) server_negotiation.cc:407] Connection header received
0926 04:02:39.867531 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:39.867531 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:39.867542 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:39.867603 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:39.868228 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:39.868954 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:39.869962 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:39.870276 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:39.870334 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:39.871744 (+ 1410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:39.871746 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:39.871746 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:02:39.871754 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:39.873404 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:39.873792 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:39.873793 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:39.873794 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:39.873847 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:39.874043 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:39.874043 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:39.874044 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:39.874191 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:39.874277 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:39.874419 (+ 142us) server_negotiation.cc:299] Negotiation successful
0926 04:02:39.874507 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260926 04:02:39.875978 22345 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb40278afbbf49d086fcebe726d0779d: Generated new TSK 0
I20260926 04:02:39.876166 22345 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 04:02:39.881372 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 22257
2026-09-26T04:02:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1305 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790395095474643-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 04:02:39 dist-test-slave-nhpw krb5kdc[22356](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 04:02:39 dist-test-slave-nhpw krb5kdc[22356](info): set up 2 sockets
Sep 26 04:02:39 dist-test-slave-nhpw krb5kdc[22356](info): commencing operation
krb5kdc: starting...
W20260926 04:02:40.904825 392 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790395095474643-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790395095474643-392-0/krb5kdc/test-user.keytab.
Sep 26 04:02:41 dist-test-slave-nhpw krb5kdc[22356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T04:02:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:02:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 04:02:41.051119 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790395095474643-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790395095474643-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790395095474643-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43019
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32801
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43019
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:02:41.129474 22369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:02:41.129714 22369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:02:41.129751 22369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:02:41.130873 22369 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:02:41.130946 22369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:02:41.130971 22369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:02:41.131001 22369 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:02:41.131034 22369 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:02:41.132505 22369 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32801
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790395095474643-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790395095474643-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43019
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43019
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22369
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790395095474643-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:02:41.132800 22369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:02:41.133026 22369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:02:41.137694 22374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:02:41.137734 22375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:02:41.137696 22377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:02:41.138095 22369 server_base.cc:1034] running on GCE node
I20260926 04:02:41.138309 22369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:02:41.138549 22369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:02:41.139706 22369 hybrid_clock.cc:648] HybridClock initialized: now 1790395361139692 us; error 44 us; skew 500 ppm
Sep 26 04:02:41 dist-test-slave-nhpw krb5kdc[22356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395361, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:02:41.142175 22369 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 04:02:41.142638 22369 webserver.cc:460] Webserver started at http://127.0.98.62:37913/ using document root <none> and password file <none>
I20260926 04:02:41.142870 22369 fs_manager.cc:359] Metadata directory not provided
I20260926 04:02:41.142937 22369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:02:41.143085 22369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:02:41.144070 22369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790395095474643-392-0/minicluster-data/master-0/data/instance:
uuid: "3d87478a1527408d8c27122676f057e5"
format_stamp: "Formatted at 2026-09-26 04:02:41 on dist-test-slave-nhpw"
server_key: "42890b1addda83019dd9bb9d3579f74f"
server_key_iv: "9c7e18e78f5cac7335eed333105f7656"
server_key_version: "encryptionkey@0"
I20260926 04:02:41.144428 22369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790395095474643-392-0/minicluster-data/master-0/wal/instance:
uuid: "3d87478a1527408d8c27122676f057e5"
format_stamp: "Formatted at 2026-09-26 04:02:41 on dist-test-slave-nhpw"
server_key: "42890b1addda83019dd9bb9d3579f74f"
server_key_iv: "9c7e18e78f5cac7335eed333105f7656"
server_key_version: "encryptionkey@0"
I20260926 04:02:41.145787 22369 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 04:02:41.146482 22384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:41.146665 22369 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 04:02:41.146763 22369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790395095474643-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790395095474643-392-0/minicluster-data/master-0/wal
uuid: "3d87478a1527408d8c27122676f057e5"
format_stamp: "Formatted at 2026-09-26 04:02:41 on dist-test-slave-nhpw"
server_key: "42890b1addda83019dd9bb9d3579f74f"
server_key_iv: "9c7e18e78f5cac7335eed333105f7656"
server_key_version: "encryptionkey@0"
I20260926 04:02:41.146848 22369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790395095474643-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790395095474643-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790395095474643-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:02:41.153142 22369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:02:41.154832 22369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:02:41.154987 22369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:02:41.159056 22369 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43019
I20260926 04:02:41.159087 22436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43019 every 8 connection(s)
I20260926 04:02:41.159430 22369 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
I20260926 04:02:41.160260 22437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:02:41.163034 22437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d87478a1527408d8c27122676f057e5: Bootstrap starting.
I20260926 04:02:41.163697 22437 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d87478a1527408d8c27122676f057e5: Neither blocks nor log segments found. Creating new log.
I20260926 04:02:41.164005 22437 log.cc:824] T 00000000000000000000000000000000 P 3d87478a1527408d8c27122676f057e5: Log is configured to *not* fsync() on all Append() calls
I20260926 04:02:41.164780 22437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d87478a1527408d8c27122676f057e5: No bootstrap required, opened a new log
I20260926 04:02:41.165433 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 22369
I20260926 04:02:41.165555 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790395095474643-392-0/minicluster-data/master-0/wal/instance
I20260926 04:02:41.165776 392 external_mini_cluster.cc:1428] Setting key 68a32130f7f0a92bb7f391b71f53dd65
I20260926 04:02:41.167044 22437 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d87478a1527408d8c27122676f057e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d87478a1527408d8c27122676f057e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43019 } }
I20260926 04:02:41.167219 22437 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d87478a1527408d8c27122676f057e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:02:41.167290 22437 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d87478a1527408d8c27122676f057e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d87478a1527408d8c27122676f057e5, State: Initialized, Role: FOLLOWER
I20260926 04:02:41.167480 22437 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d87478a1527408d8c27122676f057e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d87478a1527408d8c27122676f057e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43019 } }
I20260926 04:02:41.167580 22437 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d87478a1527408d8c27122676f057e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:02:41.167620 22437 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d87478a1527408d8c27122676f057e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:02:41.167680 22437 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d87478a1527408d8c27122676f057e5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:02:41.168462 22437 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d87478a1527408d8c27122676f057e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d87478a1527408d8c27122676f057e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43019 } }
I20260926 04:02:41.168576 22437 leader_election.cc:302] T 00000000000000000000000000000000 P 3d87478a1527408d8c27122676f057e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d87478a1527408d8c27122676f057e5; no voters:
I20260926 04:02:41.168754 22437 leader_election.cc:288] T 00000000000000000000000000000000 P 3d87478a1527408d8c27122676f057e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 04:02:41.168821 22442 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d87478a1527408d8c27122676f057e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:02:41.168987 22442 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d87478a1527408d8c27122676f057e5 [term 1 LEADER]: Becoming Leader. State: Replica: 3d87478a1527408d8c27122676f057e5, State: Running, Role: LEADER
I20260926 04:02:41.169003 22437 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d87478a1527408d8c27122676f057e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:02:41.169081 22442 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d87478a1527408d8c27122676f057e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d87478a1527408d8c27122676f057e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43019 } }
I20260926 04:02:41.169420 22442 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d87478a1527408d8c27122676f057e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d87478a1527408d8c27122676f057e5. Latest consensus state: current_term: 1 leader_uuid: "3d87478a1527408d8c27122676f057e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d87478a1527408d8c27122676f057e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43019 } } }
I20260926 04:02:41.169427 22443 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d87478a1527408d8c27122676f057e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d87478a1527408d8c27122676f057e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d87478a1527408d8c27122676f057e5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43019 } } }
I20260926 04:02:41.169559 22442 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d87478a1527408d8c27122676f057e5 [sys.catalog]: This master's current role is: LEADER
I20260926 04:02:41.169567 22443 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d87478a1527408d8c27122676f057e5 [sys.catalog]: This master's current role is: LEADER
I20260926 04:02:41.169835 22451 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:02:41.170385 22451 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:02:41.171994 22451 catalog_manager.cc:1347] Generated new cluster ID: 9ba6f43c88fd49a582eb9d9e6cfce57f
I20260926 04:02:41.172053 22451 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 04:02:41.172984 22441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:41.166277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43513 (local address 127.0.98.62:43019)
0926 04:02:41.166471 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:41.166480 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:41.166510 (+ 30us) server_negotiation.cc:407] Connection header received
0926 04:02:41.167023 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:41.167028 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:41.167080 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:41.167211 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:41.167480 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:41.169421 (+ 1941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:41.170804 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:41.171397 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:41.171535 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:41.172281 (+ 746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:41.172284 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:41.172285 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:02:41.172496 (+ 211us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:41.172525 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:02:41.172535 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:41.172611 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:41.172714 (+ 103us) server_negotiation.cc:299] Negotiation successful
0926 04:02:41.172780 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":82,"threads_started":1}
I20260926 04:02:41.173834 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 04:02:41.177901 22441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:41.175154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51819 (local address 127.0.98.62:43019)
0926 04:02:41.175193 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:41.175195 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:41.175316 (+ 121us) server_negotiation.cc:407] Connection header received
0926 04:02:41.175385 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:41.175386 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:41.175397 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:41.175456 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:41.175574 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:41.176024 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:41.176736 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:41.176941 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:41.177026 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:41.177666 (+ 640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:41.177667 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:41.177668 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:02:41.177671 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:41.177684 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:02:41.177688 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:41.177740 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:41.177805 (+ 65us) server_negotiation.cc:299] Negotiation successful
0926 04:02:41.177825 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 04:02:41.200130 22451 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:02:41.200732 22451 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:02:41.209034 22451 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d87478a1527408d8c27122676f057e5: Generated new TSK 0
I20260926 04:02:41.209223 22451 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 04:02:41.215749 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 22369
2026-09-26T04:02:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1334 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790395095474643-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 04:02:41 dist-test-slave-nhpw krb5kdc[22468](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 04:02:41 dist-test-slave-nhpw krb5kdc[22468](info): set up 2 sockets
Sep 26 04:02:41 dist-test-slave-nhpw krb5kdc[22468](info): commencing operation
krb5kdc: starting...
W20260926 04:02:42.248870 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790395095474643-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790395095474643-392-0/krb5kdc/test-user.keytab.
Sep 26 04:02:42 dist-test-slave-nhpw krb5kdc[22468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395362, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T04:02:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:02:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 04:02:42.365525 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790395095474643-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790395095474643-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790395095474643-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38049
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38049
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:02:42.444545 22481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:02:42.444775 22481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:02:42.444798 22481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:02:42.445892 22481 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:02:42.445927 22481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:02:42.445942 22481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:02:42.445955 22481 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:02:42.445967 22481 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:02:42.447352 22481 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790395095474643-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790395095474643-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38049
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790395095474643-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:02:42.447543 22481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:02:42.447705 22481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:02:42.452448 22489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:02:42.452509 22487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:02:42.452661 22481 server_base.cc:1034] running on GCE node
W20260926 04:02:42.452697 22486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:02:42.452950 22481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:02:42.453190 22481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:02:42.454332 22481 hybrid_clock.cc:648] HybridClock initialized: now 1790395362454326 us; error 32 us; skew 500 ppm
Sep 26 04:02:42 dist-test-slave-nhpw krb5kdc[22468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395362, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:02:42.464457 22481 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 04:02:42.465014 22481 webserver.cc:460] Webserver started at http://127.0.98.62:45655/ using document root <none> and password file <none>
I20260926 04:02:42.465250 22481 fs_manager.cc:359] Metadata directory not provided
I20260926 04:02:42.465318 22481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:02:42.465477 22481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:02:42.466548 22481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790395095474643-392-0/minicluster-data/master-0/data/instance:
uuid: "6addb705530744cda7e5f6f81b6eb53e"
format_stamp: "Formatted at 2026-09-26 04:02:42 on dist-test-slave-nhpw"
server_key: "ea178c079aea4faad29e85dd4671f253"
server_key_iv: "a49fde64a24810edf99fd07d78f60252"
server_key_version: "encryptionkey@0"
I20260926 04:02:42.466923 22481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790395095474643-392-0/minicluster-data/master-0/wal/instance:
uuid: "6addb705530744cda7e5f6f81b6eb53e"
format_stamp: "Formatted at 2026-09-26 04:02:42 on dist-test-slave-nhpw"
server_key: "ea178c079aea4faad29e85dd4671f253"
server_key_iv: "a49fde64a24810edf99fd07d78f60252"
server_key_version: "encryptionkey@0"
I20260926 04:02:42.468254 22481 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 04:02:42.469146 22496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:42.469336 22481 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 04:02:42.469435 22481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790395095474643-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790395095474643-392-0/minicluster-data/master-0/wal
uuid: "6addb705530744cda7e5f6f81b6eb53e"
format_stamp: "Formatted at 2026-09-26 04:02:42 on dist-test-slave-nhpw"
server_key: "ea178c079aea4faad29e85dd4671f253"
server_key_iv: "a49fde64a24810edf99fd07d78f60252"
server_key_version: "encryptionkey@0"
I20260926 04:02:42.469518 22481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790395095474643-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790395095474643-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790395095474643-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:02:42.472711 22481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:02:42.474997 22481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:02:42.475157 22481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:02:42.479180 22481 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38049
I20260926 04:02:42.479230 22548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38049 every 8 connection(s)
I20260926 04:02:42.479549 22481 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
I20260926 04:02:42.479820 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 22481
I20260926 04:02:42.480001 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790395095474643-392-0/minicluster-data/master-0/wal/instance
I20260926 04:02:42.480225 392 external_mini_cluster.cc:1428] Setting key c03da62db0c06580f8b4aff76c5bd879
I20260926 04:02:42.480305 22549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:02:42.482587 22552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:42.480777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49035 (local address 127.0.98.62:38049)
0926 04:02:42.480963 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:42.480970 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:42.480994 (+ 24us) server_negotiation.cc:407] Connection header received
0926 04:02:42.481363 (+ 369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:42.481368 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:42.481392 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:42.481543 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:42.482174 (+ 631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:42.482178 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:42.482178 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:02:42.482182 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:42.482232 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:02:42.482243 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:42.482320 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:42.482387 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 04:02:42.482433 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":82,"threads_started":1}
I20260926 04:02:42.483649 22549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6addb705530744cda7e5f6f81b6eb53e: Bootstrap starting.
I20260926 04:02:42.483662 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 04:02:42.484252 22549 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6addb705530744cda7e5f6f81b6eb53e: Neither blocks nor log segments found. Creating new log.
I20260926 04:02:42.484548 22549 log.cc:824] T 00000000000000000000000000000000 P 6addb705530744cda7e5f6f81b6eb53e: Log is configured to *not* fsync() on all Append() calls
I20260926 04:02:42.485266 22549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6addb705530744cda7e5f6f81b6eb53e: No bootstrap required, opened a new log
I20260926 04:02:42.486229 22552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:42.485181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34959 (local address 127.0.98.62:38049)
0926 04:02:42.485222 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:42.485224 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:42.485348 (+ 124us) server_negotiation.cc:407] Connection header received
0926 04:02:42.485428 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:42.485429 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:42.485433 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:42.485488 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:42.486018 (+ 530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:42.486020 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:42.486020 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 04:02:42.486024 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:42.486036 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 04:02:42.486040 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:42.486095 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:42.486151 (+ 56us) server_negotiation.cc:299] Negotiation successful
0926 04:02:42.486165 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 04:02:42.487493 22549 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6addb705530744cda7e5f6f81b6eb53e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6addb705530744cda7e5f6f81b6eb53e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38049 } }
I20260926 04:02:42.487628 22549 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6addb705530744cda7e5f6f81b6eb53e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 04:02:42.487651 22549 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6addb705530744cda7e5f6f81b6eb53e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6addb705530744cda7e5f6f81b6eb53e, State: Initialized, Role: FOLLOWER
I20260926 04:02:42.487775 22549 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6addb705530744cda7e5f6f81b6eb53e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6addb705530744cda7e5f6f81b6eb53e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38049 } }
I20260926 04:02:42.487874 22549 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6addb705530744cda7e5f6f81b6eb53e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 04:02:42.487910 22549 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6addb705530744cda7e5f6f81b6eb53e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 04:02:42.487998 22549 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6addb705530744cda7e5f6f81b6eb53e [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:02:42.488555 22549 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6addb705530744cda7e5f6f81b6eb53e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6addb705530744cda7e5f6f81b6eb53e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38049 } }
I20260926 04:02:42.488703 22549 leader_election.cc:302] T 00000000000000000000000000000000 P 6addb705530744cda7e5f6f81b6eb53e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6addb705530744cda7e5f6f81b6eb53e; no voters:
I20260926 04:02:42.488866 22549 leader_election.cc:288] T 00000000000000000000000000000000 P 6addb705530744cda7e5f6f81b6eb53e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 04:02:42.488904 22559 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6addb705530744cda7e5f6f81b6eb53e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:02:42.489118 22559 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6addb705530744cda7e5f6f81b6eb53e [term 1 LEADER]: Becoming Leader. State: Replica: 6addb705530744cda7e5f6f81b6eb53e, State: Running, Role: LEADER
I20260926 04:02:42.489123 22549 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6addb705530744cda7e5f6f81b6eb53e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:02:42.489228 22559 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6addb705530744cda7e5f6f81b6eb53e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6addb705530744cda7e5f6f81b6eb53e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38049 } }
I20260926 04:02:42.489636 22560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6addb705530744cda7e5f6f81b6eb53e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6addb705530744cda7e5f6f81b6eb53e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6addb705530744cda7e5f6f81b6eb53e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38049 } } }
I20260926 04:02:42.489643 22561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6addb705530744cda7e5f6f81b6eb53e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6addb705530744cda7e5f6f81b6eb53e. Latest consensus state: current_term: 1 leader_uuid: "6addb705530744cda7e5f6f81b6eb53e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6addb705530744cda7e5f6f81b6eb53e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38049 } } }
I20260926 04:02:42.489780 22560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6addb705530744cda7e5f6f81b6eb53e [sys.catalog]: This master's current role is: LEADER
I20260926 04:02:42.489791 22561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6addb705530744cda7e5f6f81b6eb53e [sys.catalog]: This master's current role is: LEADER
I20260926 04:02:42.490056 22568 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:02:42.490553 22568 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:02:42.492054 22568 catalog_manager.cc:1347] Generated new cluster ID: 198b735a67014902947fc047707166e2
I20260926 04:02:42.492094 22568 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 04:02:42.501916 22568 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:02:42.502090 22568 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:02:42.505491 22568 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6addb705530744cda7e5f6f81b6eb53e: Generated new TSK 0
I20260926 04:02:42.505661 22568 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 04:02:42.508075 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 22481
2026-09-26T04:02:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1292 ms)
[----------] 18 tests from AuthTokenIssuingTest (23701 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395095474643-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 04:02:42 dist-test-slave-nhpw krb5kdc[22580](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 04:02:42 dist-test-slave-nhpw krb5kdc[22580](info): set up 2 sockets
Sep 26 04:02:42 dist-test-slave-nhpw krb5kdc[22580](info): commencing operation
krb5kdc: starting...
W20260926 04:02:43.532925 392 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395095474643-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395095474643-392-0/krb5kdc/test-user.keytab.
Sep 26 04:02:43 dist-test-slave-nhpw krb5kdc[22580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T04:02:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:02:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 04:02:43.651185 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395095474643-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395095474643-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395095474643-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44889
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44889,127.0.98.61:43729,127.0.98.60:35061
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:02:43.732110 22593 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 04:02:43.732352 22593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:02:43.732383 22593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:02:43.732422 22593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:02:43.733610 22593 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:02:43.733682 22593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:02:43.733709 22593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:02:43.733740 22593 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:02:43.733762 22593 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:02:43.735291 22593 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395095474643-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395095474643-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44889,127.0.98.61:43729,127.0.98.60:35061
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395095474643-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:02:43.735564 22593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:02:43.735786 22593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:02:43.740738 22598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:02:43.740774 22599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:02:43.740738 22601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:02:43.740808 22593 server_base.cc:1034] running on GCE node
I20260926 04:02:43.741176 22593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:02:43.741468 22593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:02:43.742672 22593 hybrid_clock.cc:648] HybridClock initialized: now 1790395363742659 us; error 52 us; skew 500 ppm
Sep 26 04:02:43 dist-test-slave-nhpw krb5kdc[22580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395363, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:02:43.745353 22593 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 04:02:43.745800 22593 webserver.cc:460] Webserver started at http://127.0.98.62:44433/ using document root <none> and password file <none>
I20260926 04:02:43.746042 22593 fs_manager.cc:359] Metadata directory not provided
I20260926 04:02:43.746111 22593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:02:43.746263 22593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:02:43.747323 22593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395095474643-392-0/minicluster-data/master-0/data/instance:
uuid: "55b77c8e95a54ef6a6804e2f3f05862f"
format_stamp: "Formatted at 2026-09-26 04:02:43 on dist-test-slave-nhpw"
server_key: "c59acc48e7bbc2adec540df73316356f"
server_key_iv: "186195408c1b028723a6eed13ae30ab7"
server_key_version: "encryptionkey@0"
I20260926 04:02:43.747694 22593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395095474643-392-0/minicluster-data/master-0/wal/instance:
uuid: "55b77c8e95a54ef6a6804e2f3f05862f"
format_stamp: "Formatted at 2026-09-26 04:02:43 on dist-test-slave-nhpw"
server_key: "c59acc48e7bbc2adec540df73316356f"
server_key_iv: "186195408c1b028723a6eed13ae30ab7"
server_key_version: "encryptionkey@0"
I20260926 04:02:43.749079 22593 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 04:02:43.749815 22608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:43.750052 22593 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:02:43.750146 22593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395095474643-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395095474643-392-0/minicluster-data/master-0/wal
uuid: "55b77c8e95a54ef6a6804e2f3f05862f"
format_stamp: "Formatted at 2026-09-26 04:02:43 on dist-test-slave-nhpw"
server_key: "c59acc48e7bbc2adec540df73316356f"
server_key_iv: "186195408c1b028723a6eed13ae30ab7"
server_key_version: "encryptionkey@0"
I20260926 04:02:43.750231 22593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395095474643-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395095474643-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395095474643-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:02:43.763230 22593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:02:43.765192 22593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:02:43.765357 22593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:02:43.769279 22593 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44889
I20260926 04:02:43.769316 22660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44889 every 8 connection(s)
I20260926 04:02:43.769678 22593 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
I20260926 04:02:43.770680 22661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:02:43.772243 22661 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44889 } has no permanent_uuid. Determining permanent_uuid...
I20260926 04:02:43.775812 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 22593
I20260926 04:02:43.775904 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395095474643-392-0/minicluster-data/master-0/wal/instance
I20260926 04:02:43.776101 392 external_mini_cluster.cc:1428] Setting key efb0e662cd91e887c67e27dd193c1f45
Sep 26 04:02:43 dist-test-slave-nhpw krb5kdc[22580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395363, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 04:02:43.782451 22663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:43.772987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44889 (local address 127.0.0.1:39664)
0926 04:02:43.773151 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:43.773189 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 04:02:43.773619 (+ 430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:02:43.773873 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:02:43.773876 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:02:43.774146 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:02:43.774296 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:43.774300 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:43.775252 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:43.775253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:43.775961 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:43.775964 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:43.776503 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:43.776504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:43.776570 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:43.777159 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:02:43.777167 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:02:43.778714 (+ 1547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:02:43.780916 (+ 2202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:02:43.780922 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:02:43.780924 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:02:43.781109 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:02:43.781394 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:02:43.781395 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:02:43.781395 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:02:43.781468 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:02:43.781923 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:02:43.781927 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:02:43.781978 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 04:02:43.782046 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 04:02:43.782314 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":93,"thread_start_us":69,"threads_started":1}
I20260926 04:02:43.782457 22664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:43.773142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39664 (local address 127.0.98.62:44889)
0926 04:02:43.773255 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:43.773260 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:43.773295 (+ 35us) server_negotiation.cc:407] Connection header received
0926 04:02:43.773744 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:43.773745 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:43.773761 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:43.773837 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:43.774376 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:43.775176 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:43.776038 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:43.776382 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:43.776517 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:43.778799 (+ 2282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:43.778808 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:43.778809 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:02:43.778816 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:43.780831 (+ 2015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:43.781193 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:43.781196 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:43.781197 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:43.781308 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:43.781535 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:43.781536 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:43.781537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:43.781851 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:43.781945 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:43.782095 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 04:02:43.782245 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":58,"threads_started":1}
I20260926 04:02:43.783370 22661 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43729 } has no permanent_uuid. Determining permanent_uuid...
I20260926 04:02:43.783696 22663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:43.783587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43729 (local address 127.0.0.1:57404)
0926 04:02:43.783633 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:43.783679 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 04:02:43.783880 22609 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43729: connect: Connection refused (error 111)
I20260926 04:02:43.784282 22663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:43.784196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43729 (local address 127.0.0.1:57414)
0926 04:02:43.784244 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:43.784267 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
W20260926 04:02:43.784415 22661 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43729: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43729: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395095474643-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395095474643-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260926 04:02:43.799274 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395095474643-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395095474643-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395095474643-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395095474643-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43729
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395095474643-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44889,127.0.98.61:43729,127.0.98.60:35061
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 04:02:43.833630 22661 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43729 } attempt: 1
I20260926 04:02:43.834172 22663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:43.833985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43729 (local address 127.0.0.1:57416)
0926 04:02:43.834063 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:43.834122 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 04:02:43.834589 22663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:43.834477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43729 (local address 127.0.0.1:57424)
0926 04:02:43.834532 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:43.834563 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 04:02:43.834739 22661 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43729: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43729: connect: Connection refused (error 111)
W20260926 04:02:43.879441 22668 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 04:02:43.879654 22668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:02:43.879673 22668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:02:43.879691 22668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:02:43.880914 22668 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:02:43.880954 22668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:02:43.880970 22668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:02:43.880981 22668 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:02:43.881034 22668 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:02:43.882491 22668 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395095474643-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395095474643-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44889,127.0.98.61:43729,127.0.98.60:35061
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395095474643-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:43729
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395095474643-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22668
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395095474643-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:02:43.882745 22668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:02:43.882980 22668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:02:43.887729 22676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:02:43.887742 22674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:02:43.887831 22673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:02:43.888077 22668 server_base.cc:1034] running on GCE node
I20260926 04:02:43.888296 22668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:02:43.888535 22668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:02:43.889680 22668 hybrid_clock.cc:648] HybridClock initialized: now 1790395363889681 us; error 38 us; skew 500 ppm
Sep 26 04:02:43 dist-test-slave-nhpw krb5kdc[22580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395363, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:02:43.892248 22668 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260926 04:02:43.892750 22668 webserver.cc:460] Webserver started at http://127.0.98.61:41143/ using document root <none> and password file <none>
I20260926 04:02:43.892930 22668 fs_manager.cc:359] Metadata directory not provided
I20260926 04:02:43.892974 22668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:02:43.893100 22668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:02:43.894095 22668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395095474643-392-0/minicluster-data/master-1/data/instance:
uuid: "1d4bd3fa9cbc4735b6f50e70680ab737"
format_stamp: "Formatted at 2026-09-26 04:02:43 on dist-test-slave-nhpw"
server_key: "f7cff8591fe0f325df908a6c0b668c13"
server_key_iv: "77b408c2a2b778a97af69128f6d48a73"
server_key_version: "encryptionkey@0"
I20260926 04:02:43.894389 22668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395095474643-392-0/minicluster-data/master-1/wal/instance:
uuid: "1d4bd3fa9cbc4735b6f50e70680ab737"
format_stamp: "Formatted at 2026-09-26 04:02:43 on dist-test-slave-nhpw"
server_key: "f7cff8591fe0f325df908a6c0b668c13"
server_key_iv: "77b408c2a2b778a97af69128f6d48a73"
server_key_version: "encryptionkey@0"
I20260926 04:02:43.895637 22668 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 04:02:43.896310 22683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:43.896474 22668 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 04:02:43.896564 22668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395095474643-392-0/minicluster-data/master-1/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395095474643-392-0/minicluster-data/master-1/wal
uuid: "1d4bd3fa9cbc4735b6f50e70680ab737"
format_stamp: "Formatted at 2026-09-26 04:02:43 on dist-test-slave-nhpw"
server_key: "f7cff8591fe0f325df908a6c0b668c13"
server_key_iv: "77b408c2a2b778a97af69128f6d48a73"
server_key_version: "encryptionkey@0"
I20260926 04:02:43.896617 22668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395095474643-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395095474643-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395095474643-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:02:43.902988 22661 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43729 } attempt: 2
I20260926 04:02:43.903409 22663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:43.903267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43729 (local address 127.0.0.1:57426)
0926 04:02:43.903333 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:43.903372 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 04:02:43.903738 22663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:43.903656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43729 (local address 127.0.0.1:57442)
0926 04:02:43.903698 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:43.903721 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 04:02:43.903855 22661 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:43729: Network error: Client connection negotiation failed: client connection to 127.0.98.61:43729: connect: Connection refused (error 111)
I20260926 04:02:43.905386 22668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:02:43.907202 22668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:02:43.907369 22668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:02:43.911126 22668 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:43729
I20260926 04:02:43.911190 22735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:43729 every 8 connection(s)
I20260926 04:02:43.911514 22668 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395095474643-392-0/minicluster-data/master-1/data/info.pb
I20260926 04:02:43.912515 22736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:02:43.913420 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 22668
I20260926 04:02:43.913504 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395095474643-392-0/minicluster-data/master-1/wal/instance
I20260926 04:02:43.913686 392 external_mini_cluster.cc:1428] Setting key dde5d27335cad90ff5baa046214ca639
I20260926 04:02:43.914215 22736 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44889 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 04:02:43 dist-test-slave-nhpw krb5kdc[22580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395363, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260926 04:02:43.925230 22664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:43.914937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39680 (local address 127.0.98.62:44889)
0926 04:02:43.914996 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:43.914998 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:43.915202 (+ 204us) server_negotiation.cc:407] Connection header received
0926 04:02:43.915712 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:43.915713 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:43.915728 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:43.915778 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:43.916626 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:43.917223 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:43.918801 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:43.918999 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:43.919049 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:43.921739 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:43.921740 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:43.921740 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:02:43.921747 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:43.923606 (+ 1859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:43.924151 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:43.924152 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:43.924152 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:43.924218 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:43.924522 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:43.924524 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:43.924524 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:43.924697 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:43.924774 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:43.925075 (+ 301us) server_negotiation.cc:299] Negotiation successful
0926 04:02:43.925144 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 04:02:43.925410 22739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:43.914904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44889 (local address 127.0.0.1:39680)
0926 04:02:43.915090 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:43.915125 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 04:02:43.915601 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:02:43.915890 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:02:43.915894 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:02:43.916252 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:02:43.916516 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:43.916522 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:43.917324 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:43.917326 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:43.918700 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:43.918704 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:43.919089 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:43.919091 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:43.919243 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:43.919728 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:02:43.919738 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:02:43.921611 (+ 1873us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:02:43.923709 (+ 2098us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:02:43.923715 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:02:43.923717 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:02:43.924069 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:02:43.924301 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:02:43.924301 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:02:43.924302 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:02:43.924441 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:02:43.924764 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:02:43.924769 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:02:43.924939 (+ 170us) client_negotiation.cc:769] Sending connection context
0926 04:02:43.925054 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 04:02:43.925225 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":86,"threads_started":1}
I20260926 04:02:43.926877 22736 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43729 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 04:02:43 dist-test-slave-nhpw krb5kdc[22580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395363, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 04:02:43.934265 22739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:43.927135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43729 (local address 127.0.0.1:57446)
0926 04:02:43.927347 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:43.927358 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 04:02:43.927408 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:02:43.927541 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:02:43.927542 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:02:43.927683 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:02:43.927725 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:43.927727 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:43.928595 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:43.928596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:43.929143 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:43.929144 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:43.929641 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:43.929642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:43.929704 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:43.930195 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:02:43.930200 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:02:43.931340 (+ 1140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:02:43.933144 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:02:43.933145 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:02:43.933145 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:02:43.933320 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:02:43.933564 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:02:43.933565 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:02:43.933565 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:02:43.933623 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:02:43.933918 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:02:43.933919 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:02:43.933958 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 04:02:43.934030 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 04:02:43.934129 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 04:02:43.934296 22741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:43.927210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57446 (local address 127.0.98.61:43729)
0926 04:02:43.927361 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:43.927365 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:43.927406 (+ 41us) server_negotiation.cc:407] Connection header received
0926 04:02:43.927463 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:43.927466 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:43.927479 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:43.927542 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:43.927818 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:43.928498 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:43.929239 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:43.929552 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:43.929642 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:43.931438 (+ 1796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:43.931446 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:43.931446 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:02:43.931452 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:43.933013 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:43.933433 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:43.933436 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:43.933436 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:43.933496 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:43.933683 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:43.933684 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:43.933684 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:43.933830 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:43.933939 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:43.934059 (+ 120us) server_negotiation.cc:299] Negotiation successful
0926 04:02:43.934181 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":59,"server-negotiator.queue_time_us":124,"thread_start_us":94,"threads_started":1}
I20260926 04:02:43.934950 22736 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35061 } has no permanent_uuid. Determining permanent_uuid...
I20260926 04:02:43.935356 22739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:43.935201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35061 (local address 127.0.0.1:51970)
0926 04:02:43.935259 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:43.935332 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 04:02:43.935530 22684 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35061: connect: Connection refused (error 111)
I20260926 04:02:43.936004 22739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:43.935898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35061 (local address 127.0.0.1:51986)
0926 04:02:43.935952 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:43.935982 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 04:02:43.936141 22736 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35061: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35061: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395095474643-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395095474643-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260926 04:02:43.939791 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395095474643-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395095474643-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395095474643-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395095474643-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35061
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395095474643-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44889,127.0.98.61:43729,127.0.98.60:35061
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 04:02:43.985409 22736 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35061 } attempt: 1
I20260926 04:02:43.986045 22739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:43.985846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35061 (local address 127.0.0.1:51990)
0926 04:02:43.985929 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:43.986005 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 04:02:43.986492 22739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:43.986363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35061 (local address 127.0.0.1:51996)
0926 04:02:43.986433 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:43.986466 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260926 04:02:43.986624 22736 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35061: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35061: connect: Connection refused (error 111)
I20260926 04:02:43.995010 22661 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43729 } attempt: 3
Sep 26 04:02:43 dist-test-slave-nhpw krb5kdc[22580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395363, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 04:02:44.002107 22663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:43.995329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43729 (local address 127.0.0.1:57448)
0926 04:02:43.995389 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:43.995402 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 04:02:43.995456 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:02:43.995633 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:02:43.995634 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:02:43.995794 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:02:43.995882 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:43.995885 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:43.996591 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:43.996593 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:43.997186 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:43.997188 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:43.997601 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:43.997602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:43.997682 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:43.998172 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:02:43.998176 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:02:43.999440 (+ 1264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:02:44.001080 (+ 1640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:02:44.001081 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:02:44.001081 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:02:44.001254 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:02:44.001475 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:02:44.001476 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:02:44.001476 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:02:44.001540 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:02:44.001788 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:02:44.001789 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:02:44.001837 (+ 48us) client_negotiation.cc:769] Sending connection context
0926 04:02:44.001911 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 04:02:44.002011 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 04:02:44.002106 22741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:43.995383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57448 (local address 127.0.98.61:43729)
0926 04:02:43.995424 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:43.995426 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:43.995441 (+ 15us) server_negotiation.cc:407] Connection header received
0926 04:02:43.995541 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:43.995542 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:43.995555 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:43.995626 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:43.995987 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:43.996494 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:43.997274 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:43.997494 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:43.997588 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:43.999530 (+ 1942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:43.999532 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:43.999532 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:02:43.999540 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:44.000994 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:44.001347 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:44.001348 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:44.001349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:44.001406 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:44.001604 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:44.001606 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:44.001606 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:44.001717 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:44.001826 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:44.001936 (+ 110us) server_negotiation.cc:299] Negotiation successful
0926 04:02:44.002013 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 04:02:44.002630 22661 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35061 } has no permanent_uuid. Determining permanent_uuid...
I20260926 04:02:44.002978 22663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:44.002879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35061 (local address 127.0.0.1:52004)
0926 04:02:44.002931 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:44.002960 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 04:02:44.003266 22663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:44.003182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35061 (local address 127.0.0.1:52020)
0926 04:02:44.003224 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:44.003247 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260926 04:02:44.003368 22661 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35061: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35061: connect: Connection refused (error 111)
W20260926 04:02:44.021541 22743 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 04:02:44.021775 22743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:02:44.021840 22743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:02:44.021880 22743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:02:44.022981 22743 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:02:44.023043 22743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:02:44.023069 22743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:02:44.023100 22743 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:02:44.023128 22743 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:02:44.024696 22743 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395095474643-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395095474643-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44889,127.0.98.61:43729,127.0.98.60:35061
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395095474643-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:35061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395095474643-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22743
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395095474643-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:02:44.024962 22743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:02:44.025185 22743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:02:44.030059 22749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:02:44.030081 22748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:02:44.030061 22751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:02:44.030233 22743 server_base.cc:1034] running on GCE node
I20260926 04:02:44.030529 22743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:02:44.030830 22743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:02:44.032053 22743 hybrid_clock.cc:648] HybridClock initialized: now 1790395364032035 us; error 27 us; skew 500 ppm
Sep 26 04:02:44 dist-test-slave-nhpw krb5kdc[22580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395364, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:02:44.034507 22743 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260926 04:02:44.034507 22661 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35061 } attempt: 1
I20260926 04:02:44.034890 22663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:44.034748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35061 (local address 127.0.0.1:52036)
0926 04:02:44.034814 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:44.034860 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 04:02:44.034998 22743 webserver.cc:460] Webserver started at http://127.0.98.60:41293/ using document root <none> and password file <none>
I20260926 04:02:44.035231 22743 fs_manager.cc:359] Metadata directory not provided
I20260926 04:02:44.035275 22743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:02:44.035291 22663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:44.035178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35061 (local address 127.0.0.1:52050)
0926 04:02:44.035229 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:44.035256 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 04:02:44.035386 22661 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35061: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35061: connect: Connection refused (error 111)
I20260926 04:02:44.035413 22743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:02:44.036428 22743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395095474643-392-0/minicluster-data/master-2/data/instance:
uuid: "55c759254d1b4eeeb34a341bea6f3436"
format_stamp: "Formatted at 2026-09-26 04:02:44 on dist-test-slave-nhpw"
server_key: "316e7d481c93137a0bdc06e5e5c96a0b"
server_key_iv: "9e19b8d88c6b736305be4cfd55b178e1"
server_key_version: "encryptionkey@0"
I20260926 04:02:44.036806 22743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395095474643-392-0/minicluster-data/master-2/wal/instance:
uuid: "55c759254d1b4eeeb34a341bea6f3436"
format_stamp: "Formatted at 2026-09-26 04:02:44 on dist-test-slave-nhpw"
server_key: "316e7d481c93137a0bdc06e5e5c96a0b"
server_key_iv: "9e19b8d88c6b736305be4cfd55b178e1"
server_key_version: "encryptionkey@0"
I20260926 04:02:44.038087 22743 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 04:02:44.038755 22758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:44.038933 22743 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 04:02:44.039029 22743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395095474643-392-0/minicluster-data/master-2/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395095474643-392-0/minicluster-data/master-2/wal
uuid: "55c759254d1b4eeeb34a341bea6f3436"
format_stamp: "Formatted at 2026-09-26 04:02:44 on dist-test-slave-nhpw"
server_key: "316e7d481c93137a0bdc06e5e5c96a0b"
server_key_iv: "9e19b8d88c6b736305be4cfd55b178e1"
server_key_version: "encryptionkey@0"
I20260926 04:02:44.039106 22743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395095474643-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395095474643-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395095474643-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:02:44.051605 22743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:02:44.053715 22743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:02:44.053884 22743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:02:44.054781 22736 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35061 } attempt: 2
I20260926 04:02:44.055267 22739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:44.055115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35061 (local address 127.0.0.1:52066)
0926 04:02:44.055186 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:44.055228 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 04:02:44.055703 22739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:44.055575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35061 (local address 127.0.0.1:52074)
0926 04:02:44.055637 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:44.055672 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260926 04:02:44.055807 22736 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:35061: Network error: Client connection negotiation failed: client connection to 127.0.98.60:35061: connect: Connection refused (error 111)
I20260926 04:02:44.058216 22743 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:35061
I20260926 04:02:44.058244 22810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:35061 every 8 connection(s)
I20260926 04:02:44.058601 22743 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395095474643-392-0/minicluster-data/master-2/data/info.pb
I20260926 04:02:44.059566 22811 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:02:44.061143 22811 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44889 } has no permanent_uuid. Determining permanent_uuid...
I20260926 04:02:44.064373 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 22743
I20260926 04:02:44.064461 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790395095474643-392-0/minicluster-data/master-2/wal/instance
I20260926 04:02:44.064682 392 external_mini_cluster.cc:1428] Setting key 1b44576236b9395021f62ccfcfe34021
Sep 26 04:02:44 dist-test-slave-nhpw krb5kdc[22580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395364, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 26 04:02:44 dist-test-slave-nhpw krb5kdc[22580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 04:02:44.071236 22664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:44.061936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39684 (local address 127.0.98.62:44889)
0926 04:02:44.061995 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:44.061997 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:44.062042 (+ 45us) server_negotiation.cc:407] Connection header received
0926 04:02:44.062533 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:44.062534 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:44.062545 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:44.062605 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:44.063238 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:44.063888 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:44.065167 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:44.065396 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:44.065496 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:44.067622 (+ 2126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:44.067624 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:44.067624 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:02:44.067632 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:44.069651 (+ 2019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:44.070142 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:44.070144 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:44.070144 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:44.070204 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:44.070456 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:44.070458 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:44.070458 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:44.070577 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:44.070664 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:44.071031 (+ 367us) server_negotiation.cc:299] Negotiation successful
0926 04:02:44.071114 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 04:02:44.071380 22813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:44.061722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44889 (local address 127.0.0.1:39684)
0926 04:02:44.061920 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:44.061961 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 04:02:44.062436 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:02:44.062697 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:02:44.062701 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:02:44.062949 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:02:44.063134 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:44.063138 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:44.063978 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:44.063979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:44.065082 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:44.065086 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:44.065524 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:44.065525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:44.065660 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:44.066188 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:02:44.066196 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:02:44.067529 (+ 1333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:02:44.069764 (+ 2235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:02:44.069770 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:02:44.069772 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:02:44.070053 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:02:44.070276 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:02:44.070277 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:02:44.070277 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:02:44.070386 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:02:44.070672 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:02:44.070678 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:02:44.070811 (+ 133us) client_negotiation.cc:769] Sending connection context
0926 04:02:44.070899 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 04:02:44.071130 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":125,"threads_started":1}
I20260926 04:02:44.072002 22811 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43729 } has no permanent_uuid. Determining permanent_uuid...
I20260926 04:02:44.073891 22815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:44.065112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52086 (local address 127.0.98.60:35061)
0926 04:02:44.065426 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:44.065430 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:44.065443 (+ 13us) server_negotiation.cc:407] Connection header received
0926 04:02:44.065459 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:44.065460 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:44.065475 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:44.065552 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:44.066169 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:44.066840 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:44.067672 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:44.067962 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:44.068054 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:44.070217 (+ 2163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:44.070224 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:44.070224 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:02:44.070230 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:44.072711 (+ 2481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:44.073075 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:44.073078 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:44.073078 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:44.073140 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:44.073367 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:44.073367 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:44.073368 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:44.073524 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:44.073659 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:44.073768 (+ 109us) server_negotiation.cc:299] Negotiation successful
0926 04:02:44.073827 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":99,"threads_started":1}
Sep 26 04:02:44 dist-test-slave-nhpw krb5kdc[22580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395364, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 26 04:02:44 dist-test-slave-nhpw krb5kdc[22580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 04:02:44.079588 22813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:44.072253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:43729 (local address 127.0.0.1:57456)
0926 04:02:44.072303 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:44.072314 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 04:02:44.072357 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:02:44.072542 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:02:44.072543 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:02:44.072736 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:02:44.072803 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:44.072805 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:44.073624 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:44.073625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:44.074337 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:44.074340 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:44.074860 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:44.074861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:44.074934 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:44.075312 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:02:44.075315 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:02:44.076357 (+ 1042us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:02:44.078591 (+ 2234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:02:44.078592 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:02:44.078593 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:02:44.078751 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:02:44.078973 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:02:44.078974 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:02:44.078975 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:02:44.079036 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:02:44.079276 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:02:44.079277 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:02:44.079318 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 04:02:44.079412 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 04:02:44.079500 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 04:02:44.079598 22741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:44.072337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57456 (local address 127.0.98.61:43729)
0926 04:02:44.072422 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:44.072425 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:44.072434 (+ 9us) server_negotiation.cc:407] Connection header received
0926 04:02:44.072460 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:44.072461 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:44.072471 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:44.072541 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:44.072896 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:44.073549 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:44.074465 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:44.074770 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:44.074856 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:44.076443 (+ 1587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:44.076445 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:44.076445 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:02:44.076452 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:44.078001 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:44.078850 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:44.078851 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:44.078852 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:44.078906 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:44.079101 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:44.079102 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:44.079102 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:44.079209 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:44.079284 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:44.079439 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 04:02:44.079511 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 04:02:44.080152 22811 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35061 } has no permanent_uuid. Determining permanent_uuid...
I20260926 04:02:44.081634 22664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:44.074721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39692 (local address 127.0.98.62:44889)
0926 04:02:44.074754 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:44.074756 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:44.074772 (+ 16us) server_negotiation.cc:407] Connection header received
0926 04:02:44.074793 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:44.074794 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:44.074802 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:44.074851 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:44.075453 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:44.075896 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:44.076619 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:44.076851 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:44.076934 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:44.078938 (+ 2004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:44.078939 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:44.078939 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:02:44.078946 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:44.080603 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:44.080991 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:44.080992 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:44.080993 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:44.081040 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:44.081226 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:44.081226 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:44.081226 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:44.081307 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:44.081409 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:44.081496 (+ 87us) server_negotiation.cc:299] Negotiation successful
0926 04:02:44.081552 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
Sep 26 04:02:44 dist-test-slave-nhpw krb5kdc[22580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395364, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 26 04:02:44 dist-test-slave-nhpw krb5kdc[22580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395363, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 04:02:44.087318 22813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:44.080395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35061 (local address 127.0.0.1:52088)
0926 04:02:44.080432 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:44.080441 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 04:02:44.080471 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:02:44.080604 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:02:44.080605 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:02:44.080763 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:02:44.080787 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:44.080789 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:44.081651 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:44.081652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:44.082409 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:44.082411 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:44.082765 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:44.082765 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:44.082813 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:44.083196 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:02:44.083201 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:02:44.084371 (+ 1170us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:02:44.086281 (+ 1910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:02:44.086282 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:02:44.086282 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:02:44.086439 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:02:44.086693 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:02:44.086694 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:02:44.086694 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:02:44.086747 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:02:44.086995 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:02:44.086995 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:02:44.087032 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 04:02:44.087101 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 04:02:44.087191 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260926 04:02:44.087327 22815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:44.080454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52088 (local address 127.0.98.60:35061)
0926 04:02:44.080493 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:44.080495 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:44.080503 (+ 8us) server_negotiation.cc:407] Connection header received
0926 04:02:44.080523 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:44.080524 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:44.080533 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:44.080602 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:44.080860 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:44.081567 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:44.082480 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:44.082683 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:44.082765 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:44.084470 (+ 1705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:44.084471 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:44.084472 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:02:44.084479 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:44.086176 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:44.086547 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:44.086548 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:44.086548 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:44.086599 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:44.086827 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:44.086828 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:44.086828 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:44.086931 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:44.087002 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:44.087126 (+ 124us) server_negotiation.cc:299] Negotiation successful
0926 04:02:44.087236 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":21}
I20260926 04:02:44.089217 22811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55c759254d1b4eeeb34a341bea6f3436: Bootstrap starting.
I20260926 04:02:44.090050 22741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:44.082370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57472 (local address 127.0.98.61:43729)
0926 04:02:44.082404 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:44.082406 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:44.082414 (+ 8us) server_negotiation.cc:407] Connection header received
0926 04:02:44.082436 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:44.082437 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:44.082445 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:44.082489 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:44.082940 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:44.083369 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:44.084088 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:44.084426 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:44.084486 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:44.086859 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:44.086860 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:44.086861 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:02:44.086867 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:44.088956 (+ 2089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:44.089297 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:44.089298 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:44.089299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:44.089355 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:44.089574 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:44.089575 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:44.089575 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:44.089683 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:44.089829 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:44.089934 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 04:02:44.089993 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 04:02:44.090250 22811 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55c759254d1b4eeeb34a341bea6f3436: Neither blocks nor log segments found. Creating new log.
I20260926 04:02:44.090631 22811 log.cc:824] T 00000000000000000000000000000000 P 55c759254d1b4eeeb34a341bea6f3436: Log is configured to *not* fsync() on all Append() calls
I20260926 04:02:44.091511 22811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55c759254d1b4eeeb34a341bea6f3436: No bootstrap required, opened a new log
I20260926 04:02:44.091670 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 04:02:44.093238 22811 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55c759254d1b4eeeb34a341bea6f3436 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55b77c8e95a54ef6a6804e2f3f05862f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44889 } } peers { permanent_uuid: "1d4bd3fa9cbc4735b6f50e70680ab737" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43729 } } peers { permanent_uuid: "55c759254d1b4eeeb34a341bea6f3436" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35061 } }
I20260926 04:02:44.093412 22811 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55c759254d1b4eeeb34a341bea6f3436 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55c759254d1b4eeeb34a341bea6f3436, State: Initialized, Role: FOLLOWER
I20260926 04:02:44.093497 22811 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55c759254d1b4eeeb34a341bea6f3436 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55b77c8e95a54ef6a6804e2f3f05862f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44889 } } peers { permanent_uuid: "1d4bd3fa9cbc4735b6f50e70680ab737" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43729 } } peers { permanent_uuid: "55c759254d1b4eeeb34a341bea6f3436" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35061 } }
I20260926 04:02:44.092621 22635 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "55b77c8e95a54ef6a6804e2f3f05862f"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39692
I20260926 04:02:44.093815 22811 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55c759254d1b4eeeb34a341bea6f3436 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:02:44.095156 22817 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55c759254d1b4eeeb34a341bea6f3436 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55b77c8e95a54ef6a6804e2f3f05862f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44889 } } peers { permanent_uuid: "1d4bd3fa9cbc4735b6f50e70680ab737" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43729 } } peers { permanent_uuid: "55c759254d1b4eeeb34a341bea6f3436" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35061 } } }
I20260926 04:02:44.095448 22817 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55c759254d1b4eeeb34a341bea6f3436 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 04:02:44.095671 22828 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 55c759254d1b4eeeb34a341bea6f3436: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 04:02:44.095721 22828 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 04:02:44.095935 22743 master_runner.cc:186] Error getting master registration for 127.0.98.62:44889: OK, instance_id { permanent_uuid: "55b77c8e95a54ef6a6804e2f3f05862f" instance_seqno: 1790395363768075 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 04:02:44.098764 22743 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 04:02:44.110486 22661 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35061 } attempt: 2
Sep 26 04:02:44 dist-test-slave-nhpw krb5kdc[22580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395363, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 04:02:44.117753 22815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:44.110917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52100 (local address 127.0.98.60:35061)
0926 04:02:44.110957 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:44.110960 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:44.111008 (+ 48us) server_negotiation.cc:407] Connection header received
0926 04:02:44.111084 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:44.111085 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:44.111095 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:44.111174 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:44.111492 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:44.111968 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:44.112679 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:44.112886 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:44.112977 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:44.115021 (+ 2044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:44.115022 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:44.115022 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:02:44.115029 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:44.116519 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:44.116920 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:44.116922 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:44.116922 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:44.116983 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:44.117182 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:44.117183 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:44.117184 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:44.117332 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:44.117415 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:44.117568 (+ 153us) server_negotiation.cc:299] Negotiation successful
0926 04:02:44.117646 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 04:02:44.117755 22663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:44.110855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35061 (local address 127.0.0.1:52100)
0926 04:02:44.110946 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:44.110959 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 04:02:44.111027 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:02:44.111181 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:02:44.111182 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:02:44.111362 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:02:44.111406 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:44.111407 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:44.112056 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:44.112057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:44.112572 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:44.112573 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:44.112976 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:44.112977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:44.113036 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:44.113506 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:02:44.113511 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:02:44.114899 (+ 1388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:02:44.116659 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:02:44.116660 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:02:44.116661 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:02:44.116843 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:02:44.117052 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:02:44.117053 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:02:44.117053 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:02:44.117111 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:02:44.117407 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:02:44.117408 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:02:44.117455 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 04:02:44.117544 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 04:02:44.117646 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 04:02:44.119652 22661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55b77c8e95a54ef6a6804e2f3f05862f: Bootstrap starting.
I20260926 04:02:44.120267 22661 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55b77c8e95a54ef6a6804e2f3f05862f: Neither blocks nor log segments found. Creating new log.
I20260926 04:02:44.120570 22661 log.cc:824] T 00000000000000000000000000000000 P 55b77c8e95a54ef6a6804e2f3f05862f: Log is configured to *not* fsync() on all Append() calls
I20260926 04:02:44.121340 22661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55b77c8e95a54ef6a6804e2f3f05862f: No bootstrap required, opened a new log
I20260926 04:02:44.122351 22661 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55b77c8e95a54ef6a6804e2f3f05862f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55b77c8e95a54ef6a6804e2f3f05862f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44889 } } peers { permanent_uuid: "1d4bd3fa9cbc4735b6f50e70680ab737" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43729 } } peers { permanent_uuid: "55c759254d1b4eeeb34a341bea6f3436" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35061 } }
I20260926 04:02:44.122486 22661 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55b77c8e95a54ef6a6804e2f3f05862f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55b77c8e95a54ef6a6804e2f3f05862f, State: Initialized, Role: FOLLOWER
I20260926 04:02:44.122575 22661 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55b77c8e95a54ef6a6804e2f3f05862f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55b77c8e95a54ef6a6804e2f3f05862f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44889 } } peers { permanent_uuid: "1d4bd3fa9cbc4735b6f50e70680ab737" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43729 } } peers { permanent_uuid: "55c759254d1b4eeeb34a341bea6f3436" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35061 } }
I20260926 04:02:44.122844 22831 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55b77c8e95a54ef6a6804e2f3f05862f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55b77c8e95a54ef6a6804e2f3f05862f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44889 } } peers { permanent_uuid: "1d4bd3fa9cbc4735b6f50e70680ab737" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43729 } } peers { permanent_uuid: "55c759254d1b4eeeb34a341bea6f3436" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35061 } } }
I20260926 04:02:44.122898 22661 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55b77c8e95a54ef6a6804e2f3f05862f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:02:44.122931 22831 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55b77c8e95a54ef6a6804e2f3f05862f [sys.catalog]: This master's current role is: FOLLOWER
I20260926 04:02:44.123036 22635 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55b77c8e95a54ef6a6804e2f3f05862f [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 04:02:44.123119 22635 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55b77c8e95a54ef6a6804e2f3f05862f [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:02:44.123893 22635 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55b77c8e95a54ef6a6804e2f3f05862f [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55b77c8e95a54ef6a6804e2f3f05862f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44889 } } peers { permanent_uuid: "1d4bd3fa9cbc4735b6f50e70680ab737" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43729 } } peers { permanent_uuid: "55c759254d1b4eeeb34a341bea6f3436" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35061 } }
I20260926 04:02:44.124423 22785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "55b77c8e95a54ef6a6804e2f3f05862f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "55c759254d1b4eeeb34a341bea6f3436"
I20260926 04:02:44.125056 22785 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55c759254d1b4eeeb34a341bea6f3436 [term 0 FOLLOWER]: Advancing to term 1
W20260926 04:02:44.125387 22842 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 55b77c8e95a54ef6a6804e2f3f05862f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 04:02:44.125447 22842 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 04:02:44.125909 22785 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 55c759254d1b4eeeb34a341bea6f3436 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55b77c8e95a54ef6a6804e2f3f05862f in term 1.
I20260926 04:02:44.126444 22609 leader_election.cc:302] T 00000000000000000000000000000000 P 55b77c8e95a54ef6a6804e2f3f05862f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 55b77c8e95a54ef6a6804e2f3f05862f, 55c759254d1b4eeeb34a341bea6f3436; no voters:
I20260926 04:02:44.126600 22831 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55b77c8e95a54ef6a6804e2f3f05862f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:02:44.126744 22831 raft_consensus.cc:686] T 00000000000000000000000000000000 P 55b77c8e95a54ef6a6804e2f3f05862f [term 1 LEADER]: Becoming Leader. State: Replica: 55b77c8e95a54ef6a6804e2f3f05862f, State: Running, Role: LEADER
I20260926 04:02:44.126828 22831 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55b77c8e95a54ef6a6804e2f3f05862f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55b77c8e95a54ef6a6804e2f3f05862f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44889 } } peers { permanent_uuid: "1d4bd3fa9cbc4735b6f50e70680ab737" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43729 } } peers { permanent_uuid: "55c759254d1b4eeeb34a341bea6f3436" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35061 } }
I20260926 04:02:44.127267 22844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55b77c8e95a54ef6a6804e2f3f05862f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55b77c8e95a54ef6a6804e2f3f05862f. Latest consensus state: current_term: 1 leader_uuid: "55b77c8e95a54ef6a6804e2f3f05862f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55b77c8e95a54ef6a6804e2f3f05862f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44889 } } peers { permanent_uuid: "1d4bd3fa9cbc4735b6f50e70680ab737" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43729 } } peers { permanent_uuid: "55c759254d1b4eeeb34a341bea6f3436" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35061 } } }
I20260926 04:02:44.127472 22844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55b77c8e95a54ef6a6804e2f3f05862f [sys.catalog]: This master's current role is: LEADER
I20260926 04:02:44.128574 22846 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:02:44.128935 22846 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:02:44.127660 22593 master_runner.cc:186] Error getting master registration for 127.0.98.61:43729: OK, instance_id { permanent_uuid: "1d4bd3fa9cbc4735b6f50e70680ab737" instance_seqno: 1790395363910155 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 04:02:44.129981 22593 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 04:02:44.130036 22635 leader_election.cc:288] T 00000000000000000000000000000000 P 55b77c8e95a54ef6a6804e2f3f05862f [CANDIDATE]: Term 1 election: Requested vote from peers 1d4bd3fa9cbc4735b6f50e70680ab737 (127.0.98.61:43729), 55c759254d1b4eeeb34a341bea6f3436 (127.0.98.60:35061)
I20260926 04:02:44.131525 22785 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 55c759254d1b4eeeb34a341bea6f3436 [term 1 FOLLOWER]: Refusing update from remote peer 55b77c8e95a54ef6a6804e2f3f05862f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:02:44.131930 22844 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 55b77c8e95a54ef6a6804e2f3f05862f [LEADER]: Connected to new peer: Peer: permanent_uuid: "55c759254d1b4eeeb34a341bea6f3436" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:02:44.133208 22817 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55c759254d1b4eeeb34a341bea6f3436 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55b77c8e95a54ef6a6804e2f3f05862f. Latest consensus state: current_term: 1 leader_uuid: "55b77c8e95a54ef6a6804e2f3f05862f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55b77c8e95a54ef6a6804e2f3f05862f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44889 } } peers { permanent_uuid: "1d4bd3fa9cbc4735b6f50e70680ab737" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43729 } } peers { permanent_uuid: "55c759254d1b4eeeb34a341bea6f3436" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35061 } } }
I20260926 04:02:44.133297 22817 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55c759254d1b4eeeb34a341bea6f3436 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 04:02:44.134471 22844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55b77c8e95a54ef6a6804e2f3f05862f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "55b77c8e95a54ef6a6804e2f3f05862f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55b77c8e95a54ef6a6804e2f3f05862f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44889 } } peers { permanent_uuid: "1d4bd3fa9cbc4735b6f50e70680ab737" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43729 } } peers { permanent_uuid: "55c759254d1b4eeeb34a341bea6f3436" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35061 } } }
I20260926 04:02:44.134548 22844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55b77c8e95a54ef6a6804e2f3f05862f [sys.catalog]: This master's current role is: LEADER
I20260926 04:02:44.134865 22846 catalog_manager.cc:1347] Generated new cluster ID: afe532907d42458eb4e864e10677f46a
I20260926 04:02:44.134917 22846 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 04:02:44.135125 22817 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55c759254d1b4eeeb34a341bea6f3436 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "55b77c8e95a54ef6a6804e2f3f05862f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55b77c8e95a54ef6a6804e2f3f05862f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44889 } } peers { permanent_uuid: "1d4bd3fa9cbc4735b6f50e70680ab737" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43729 } } peers { permanent_uuid: "55c759254d1b4eeeb34a341bea6f3436" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35061 } } }
I20260926 04:02:44.135195 22817 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55c759254d1b4eeeb34a341bea6f3436 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 04:02:44.135461 22853 mvcc.cc:205] Tried to move back new op lower bound from 7333459411480776704 to 7333459411464433664. Current Snapshot: MvccSnapshot[applied={T|T < 7333459411480776704}]
I20260926 04:02:44.146939 22736 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35061 } attempt: 3
Sep 26 04:02:44 dist-test-slave-nhpw krb5kdc[22580](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395363, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 04:02:44.154428 22846 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:02:44.155283 22846 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:02:44.158380 22741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:44.136867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57478 (local address 127.0.98.61:43729)
0926 04:02:44.136912 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:44.136915 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:44.148773 (+ 11858us) server_negotiation.cc:407] Connection header received
0926 04:02:44.148881 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:44.148883 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:44.148893 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:44.148919 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:44.149546 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:44.150243 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:44.151246 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:44.151542 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:44.151580 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:44.153106 (+ 1526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:44.153108 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:44.153108 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:02:44.153115 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:44.154840 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:44.155491 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:44.155493 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:44.155493 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:44.155552 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:44.157602 (+ 2050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:44.157604 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:44.157604 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:44.157726 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:44.157857 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:44.158189 (+ 332us) server_negotiation.cc:299] Negotiation successful
0926 04:02:44.158274 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 04:02:44.159204 22846 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 55b77c8e95a54ef6a6804e2f3f05862f: Generated new TSK 0
I20260926 04:02:44.159432 22846 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 04:02:44.159449 22739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:44.147181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:35061 (local address 127.0.0.1:52122)
0926 04:02:44.147244 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:44.147254 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 04:02:44.147290 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:02:44.147774 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:02:44.147776 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:02:44.147983 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:02:44.148032 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:44.148034 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:44.149786 (+ 1752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:44.149787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:44.150568 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:44.150571 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:44.151457 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:44.151458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:44.151520 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:44.152013 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:02:44.152019 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:02:44.154039 (+ 2020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:02:44.157394 (+ 3355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:02:44.157395 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:02:44.157395 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:02:44.157610 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:02:44.158037 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:02:44.158038 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:02:44.158038 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:02:44.158112 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:02:44.159167 (+ 1055us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:02:44.159168 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:02:44.159225 (+ 57us) client_negotiation.cc:769] Sending connection context
0926 04:02:44.159272 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 04:02:44.159364 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260926 04:02:44.160579 22664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:44.131748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39696 (local address 127.0.98.62:44889)
0926 04:02:44.131786 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:44.131788 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:44.142602 (+ 10814us) server_negotiation.cc:407] Connection header received
0926 04:02:44.142667 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:44.142668 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:44.142693 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:44.142740 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:44.149411 (+ 6671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:44.150094 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:44.152106 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:44.152397 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:44.152442 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:44.155678 (+ 3236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:44.155679 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:44.155679 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:02:44.155687 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:44.157379 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:44.158914 (+ 1535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:44.158915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:44.158915 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:44.158976 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:44.159191 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:44.159192 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:44.159192 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:44.159315 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:44.159394 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:44.160376 (+ 982us) server_negotiation.cc:299] Negotiation successful
0926 04:02:44.160469 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 04:02:44.160799 22859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:44.147387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52122 (local address 127.0.98.60:35061)
0926 04:02:44.147567 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:44.147569 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:44.147576 (+ 7us) server_negotiation.cc:407] Connection header received
0926 04:02:44.147624 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:44.147625 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:44.147636 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:44.148122 (+ 486us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:44.148189 (+ 67us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:44.149671 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:44.150709 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:44.151330 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:44.151712 (+ 382us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:44.154400 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:44.154401 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:44.154429 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:02:44.154435 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:44.157278 (+ 2843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:44.157754 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:44.157756 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:44.157756 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:44.157934 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:44.158348 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:44.158348 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:44.158349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:44.159064 (+ 715us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:44.160008 (+ 944us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:44.160089 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 04:02:44.160231 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":71,"threads_started":1}
I20260926 04:02:44.162971 22736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d4bd3fa9cbc4735b6f50e70680ab737: Bootstrap starting.
I20260926 04:02:44.163657 22736 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d4bd3fa9cbc4735b6f50e70680ab737: Neither blocks nor log segments found. Creating new log.
I20260926 04:02:44.163957 22736 log.cc:824] T 00000000000000000000000000000000 P 1d4bd3fa9cbc4735b6f50e70680ab737: Log is configured to *not* fsync() on all Append() calls
I20260926 04:02:44.164197 22815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:44.137051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52114 (local address 127.0.98.60:35061)
0926 04:02:44.137093 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:44.137095 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:44.151176 (+ 14081us) server_negotiation.cc:407] Connection header received
0926 04:02:44.151269 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:44.151270 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:44.151279 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:44.151309 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:44.153908 (+ 2599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:44.154618 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:44.155947 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:44.156253 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:44.156297 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:44.160311 (+ 4014us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:44.160312 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:44.160313 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:02:44.160320 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:44.162533 (+ 2213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:44.162943 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:44.162944 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:44.162945 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:44.163018 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:44.163298 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:44.163299 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:44.163299 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:44.163417 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:44.163491 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:44.163683 (+ 192us) server_negotiation.cc:299] Negotiation successful
0926 04:02:44.164105 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 04:02:44.168932 22736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d4bd3fa9cbc4735b6f50e70680ab737: No bootstrap required, opened a new log
I20260926 04:02:44.171238 22736 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d4bd3fa9cbc4735b6f50e70680ab737 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55b77c8e95a54ef6a6804e2f3f05862f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44889 } } peers { permanent_uuid: "1d4bd3fa9cbc4735b6f50e70680ab737" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43729 } } peers { permanent_uuid: "55c759254d1b4eeeb34a341bea6f3436" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35061 } }
I20260926 04:02:44.171391 22736 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d4bd3fa9cbc4735b6f50e70680ab737 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d4bd3fa9cbc4735b6f50e70680ab737, State: Initialized, Role: FOLLOWER
I20260926 04:02:44.171470 22736 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d4bd3fa9cbc4735b6f50e70680ab737 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55b77c8e95a54ef6a6804e2f3f05862f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44889 } } peers { permanent_uuid: "1d4bd3fa9cbc4735b6f50e70680ab737" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43729 } } peers { permanent_uuid: "55c759254d1b4eeeb34a341bea6f3436" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35061 } }
I20260926 04:02:44.171797 22736 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d4bd3fa9cbc4735b6f50e70680ab737 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:02:44.171876 22709 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d4bd3fa9cbc4735b6f50e70680ab737 [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:02:44.172775 22709 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1d4bd3fa9cbc4735b6f50e70680ab737 [term 1 FOLLOWER]: Refusing update from remote peer 55b77c8e95a54ef6a6804e2f3f05862f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:02:44.172950 22710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "55b77c8e95a54ef6a6804e2f3f05862f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1d4bd3fa9cbc4735b6f50e70680ab737"
I20260926 04:02:44.173671 22710 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1d4bd3fa9cbc4735b6f50e70680ab737 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 55b77c8e95a54ef6a6804e2f3f05862f in term 1.
I20260926 04:02:44.174130 22831 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 55b77c8e95a54ef6a6804e2f3f05862f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d4bd3fa9cbc4735b6f50e70680ab737" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 04:02:44.177824 22873 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1d4bd3fa9cbc4735b6f50e70680ab737: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 04:02:44.177891 22873 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 04:02:44.182610 22874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d4bd3fa9cbc4735b6f50e70680ab737 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "55b77c8e95a54ef6a6804e2f3f05862f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55b77c8e95a54ef6a6804e2f3f05862f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44889 } } peers { permanent_uuid: "1d4bd3fa9cbc4735b6f50e70680ab737" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43729 } } peers { permanent_uuid: "55c759254d1b4eeeb34a341bea6f3436" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35061 } } }
I20260926 04:02:44.182758 22874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d4bd3fa9cbc4735b6f50e70680ab737 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 04:02:44.182848 22874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d4bd3fa9cbc4735b6f50e70680ab737 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55b77c8e95a54ef6a6804e2f3f05862f. Latest consensus state: current_term: 1 leader_uuid: "55b77c8e95a54ef6a6804e2f3f05862f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55b77c8e95a54ef6a6804e2f3f05862f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44889 } } peers { permanent_uuid: "1d4bd3fa9cbc4735b6f50e70680ab737" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43729 } } peers { permanent_uuid: "55c759254d1b4eeeb34a341bea6f3436" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35061 } } }
I20260926 04:02:44.182947 22874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d4bd3fa9cbc4735b6f50e70680ab737 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 04:02:44.183012 22874 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d4bd3fa9cbc4735b6f50e70680ab737 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "55b77c8e95a54ef6a6804e2f3f05862f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55b77c8e95a54ef6a6804e2f3f05862f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44889 } } peers { permanent_uuid: "1d4bd3fa9cbc4735b6f50e70680ab737" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43729 } } peers { permanent_uuid: "55c759254d1b4eeeb34a341bea6f3436" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35061 } } }
I20260926 04:02:44.183103 22874 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d4bd3fa9cbc4735b6f50e70680ab737 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 04:02:44.183693 22875 mvcc.cc:205] Tried to move back new op lower bound from 7333459411591491584 to 7333459411464433664. Current Snapshot: MvccSnapshot[applied={T|T < 7333459411480776704}]
W20260926 04:02:44.193537 22741 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:02:44.188933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57484 (local address 127.0.98.61:43729)
0926 04:02:44.188972 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:44.188974 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:44.192843 (+ 3869us) server_negotiation.cc:407] Connection header received
0926 04:02:44.192952 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:44.192954 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:44.192965 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:44.192994 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:44.193441 (+ 447us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57484: BlockingRecv error: recv got EOF from 127.0.0.1:57484 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260926 04:02:44.194432 22664 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:02:44.189165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39708 (local address 127.0.98.62:44889)
0926 04:02:44.189201 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:44.189203 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:44.193791 (+ 4588us) server_negotiation.cc:407] Connection header received
0926 04:02:44.193892 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:44.193893 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:44.193903 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:44.193954 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:44.194371 (+ 417us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39708: BlockingRecv error: recv got EOF from 127.0.0.1:39708 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260926 04:02:44.195354 22859 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:02:44.189492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52134 (local address 127.0.98.60:35061)
0926 04:02:44.189532 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:44.189534 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:44.194660 (+ 5126us) server_negotiation.cc:407] Connection header received
0926 04:02:44.194818 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:44.194819 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:44.194830 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:44.194860 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:44.195299 (+ 439us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52134: BlockingRecv error: recv got EOF from 127.0.0.1:52134 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260926 04:02:44.197808 22741 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:02:44.197001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57492 (local address 127.0.98.61:43729)
0926 04:02:44.197037 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:44.197039 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:44.197283 (+ 244us) server_negotiation.cc:407] Connection header received
0926 04:02:44.197384 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:44.197385 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:44.197401 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:44.197430 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:44.197719 (+ 289us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57492: BlockingRecv error: recv got EOF from 127.0.0.1:57492 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260926 04:02:44.205323 22741 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:02:44.200961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57494 (local address 127.0.98.61:43729)
0926 04:02:44.200991 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:44.200993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:44.204834 (+ 3841us) server_negotiation.cc:407] Connection header received
0926 04:02:44.204932 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:44.204933 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:44.204944 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:44.204972 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:44.205279 (+ 307us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57494: BlockingRecv error: recv got EOF from 127.0.0.1:57494 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260926 04:02:44.209113 22741 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:02:44.208385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57500 (local address 127.0.98.61:43729)
0926 04:02:44.208417 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:44.208419 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:44.208622 (+ 203us) server_negotiation.cc:407] Connection header received
0926 04:02:44.208738 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:44.208739 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:44.208749 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:44.208792 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:44.209072 (+ 280us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57500: BlockingRecv error: recv got EOF from 127.0.0.1:57500 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260926 04:02:44.214836 22741 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:02:44.214105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57508 (local address 127.0.98.61:43729)
0926 04:02:44.214135 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:44.214137 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:44.214392 (+ 255us) server_negotiation.cc:407] Connection header received
0926 04:02:44.214480 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:44.214481 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:44.214490 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:44.214517 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:44.214800 (+ 283us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57508: BlockingRecv error: recv got EOF from 127.0.0.1:57508 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260926 04:02:44.224792 22741 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:02:44.223972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57516 (local address 127.0.98.61:43729)
0926 04:02:44.224012 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:44.224013 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:44.224262 (+ 249us) server_negotiation.cc:407] Connection header received
0926 04:02:44.224360 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:44.224361 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:44.224370 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:44.224397 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:44.224738 (+ 341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57516: BlockingRecv error: recv got EOF from 127.0.0.1:57516 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260926 04:02:44.243214 22741 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:02:44.242081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57528 (local address 127.0.98.61:43729)
0926 04:02:44.242143 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:44.242145 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:44.242571 (+ 426us) server_negotiation.cc:407] Connection header received
0926 04:02:44.242681 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:44.242682 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:44.242694 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:44.242724 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:44.243159 (+ 435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57528: BlockingRecv error: recv got EOF from 127.0.0.1:57528 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260926 04:02:44.277529 22741 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:02:44.276601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57536 (local address 127.0.98.61:43729)
0926 04:02:44.276666 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:44.276669 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:44.276940 (+ 271us) server_negotiation.cc:407] Connection header received
0926 04:02:44.277042 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:44.277043 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:44.277055 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:44.277084 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:44.277466 (+ 382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57536: BlockingRecv error: recv got EOF from 127.0.0.1:57536 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260926 04:02:44.343739 22741 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:02:44.342959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57544 (local address 127.0.98.61:43729)
0926 04:02:44.343031 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:44.343034 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:44.343158 (+ 124us) server_negotiation.cc:407] Connection header received
0926 04:02:44.343241 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:44.343242 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:44.343253 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:44.343321 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:44.343676 (+ 355us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57544: BlockingRecv error: recv got EOF from 127.0.0.1:57544 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260926 04:02:44.473822 22741 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:02:44.473039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57560 (local address 127.0.98.61:43729)
0926 04:02:44.473126 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:44.473128 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:44.473229 (+ 101us) server_negotiation.cc:407] Connection header received
0926 04:02:44.473309 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:44.473310 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:44.473323 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:44.473378 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:44.473719 (+ 341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57560: BlockingRecv error: recv got EOF from 127.0.0.1:57560 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260926 04:02:44.731741 22741 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:02:44.731050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57572 (local address 127.0.98.61:43729)
0926 04:02:44.731132 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:44.731134 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:44.731145 (+ 11us) server_negotiation.cc:407] Connection header received
0926 04:02:44.731221 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:44.731223 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:44.731236 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:44.731307 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:44.731653 (+ 346us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57572: BlockingRecv error: recv got EOF from 127.0.0.1:57572 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 04:02:44.740190 22940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55b77c8e95a54ef6a6804e2f3f05862f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "55b77c8e95a54ef6a6804e2f3f05862f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "55b77c8e95a54ef6a6804e2f3f05862f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44889 } } peers { permanent_uuid: "1d4bd3fa9cbc4735b6f50e70680ab737" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 43729 } } peers { permanent_uuid: "55c759254d1b4eeeb34a341bea6f3436" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 35061 } } }
I20260926 04:02:44.740377 22940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55b77c8e95a54ef6a6804e2f3f05862f [sys.catalog]: This master's current role is: LEADER
I20260926 04:02:45.096155 22828 catalog_manager.cc:1259] Loaded cluster ID: afe532907d42458eb4e864e10677f46a
I20260926 04:02:45.096238 22828 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 55c759254d1b4eeeb34a341bea6f3436: loading cluster ID for follower catalog manager: success
I20260926 04:02:45.097080 22828 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 55c759254d1b4eeeb34a341bea6f3436: acquiring CA information for follower catalog manager: success
I20260926 04:02:45.097388 22828 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 55c759254d1b4eeeb34a341bea6f3436: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 04:02:45.178324 22873 catalog_manager.cc:1259] Loaded cluster ID: afe532907d42458eb4e864e10677f46a
I20260926 04:02:45.178398 22873 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1d4bd3fa9cbc4735b6f50e70680ab737: loading cluster ID for follower catalog manager: success
I20260926 04:02:45.179173 22873 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1d4bd3fa9cbc4735b6f50e70680ab737: acquiring CA information for follower catalog manager: success
I20260926 04:02:45.179418 22873 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1d4bd3fa9cbc4735b6f50e70680ab737: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 04:02:45.247812 22947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:45.245255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57576 (local address 127.0.98.61:43729)
0926 04:02:45.245390 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:45.245391 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:45.245400 (+ 9us) server_negotiation.cc:407] Connection header received
0926 04:02:45.245427 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:45.245428 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:45.245483 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:45.245561 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 04:02:45.245771 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:45.246273 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:45.247023 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:45.247222 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:45.247281 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:45.247427 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 04:02:45.247534 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 04:02:45.247594 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:45.247656 (+ 62us) server_negotiation.cc:299] Negotiation successful
0926 04:02:45.247674 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":73,"threads_started":1}
I20260926 04:02:45.252079 22953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:45.249743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52146 (local address 127.0.98.60:35061)
0926 04:02:45.249862 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:45.249864 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:45.249907 (+ 43us) server_negotiation.cc:407] Connection header received
0926 04:02:45.249974 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:45.249975 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:45.250007 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:45.250074 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 04:02:45.250158 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:45.250629 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:45.251328 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:45.251524 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:45.251581 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:45.251694 (+ 113us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 04:02:45.251795 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 04:02:45.251852 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:45.251941 (+ 89us) server_negotiation.cc:299] Negotiation successful
0926 04:02:45.251958 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":60,"threads_started":1}
I20260926 04:02:45.256413 22947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:45.253858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57578 (local address 127.0.98.61:43729)
0926 04:02:45.253901 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:45.253902 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:45.254018 (+ 116us) server_negotiation.cc:407] Connection header received
0926 04:02:45.254127 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:45.254127 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:45.254139 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:45.254193 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 04:02:45.254305 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:45.254917 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:45.255685 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:45.255893 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:45.255979 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:45.256137 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 04:02:45.256176 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 04:02:45.256241 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:45.256302 (+ 61us) server_negotiation.cc:299] Negotiation successful
0926 04:02:45.256323 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 04:02:45.257489 22953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:45.254081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52158 (local address 127.0.98.60:35061)
0926 04:02:45.254116 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:45.254118 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:45.254426 (+ 308us) server_negotiation.cc:407] Connection header received
0926 04:02:45.254453 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:45.254454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:45.254464 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:45.254518 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 04:02:45.254608 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:45.255058 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:45.256775 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:45.257015 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:45.257068 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:45.257235 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 04:02:45.257271 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 04:02:45.257320 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:45.257388 (+ 68us) server_negotiation.cc:299] Negotiation successful
0926 04:02:45.257406 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 04:02:45.257755 22960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:45.253910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39716 (local address 127.0.98.62:44889)
0926 04:02:45.254071 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:45.254074 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:45.254081 (+ 7us) server_negotiation.cc:407] Connection header received
0926 04:02:45.254129 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:45.254130 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:45.254184 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:45.254275 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 04:02:45.255151 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:45.255652 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:45.256840 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:45.257036 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:45.257122 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:45.257369 (+ 247us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 04:02:45.257487 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 04:02:45.257572 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:45.257637 (+ 65us) server_negotiation.cc:299] Negotiation successful
0926 04:02:45.257662 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":68,"threads_started":1}
I20260926 04:02:45.259548 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 22593
I20260926 04:02:45.264875 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 22668
I20260926 04:02:45.269694 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 22743
2026-09-26T04:02:45Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2762 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395095474643-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 04:02:45 dist-test-slave-nhpw krb5kdc[22964](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 04:02:45 dist-test-slave-nhpw krb5kdc[22964](info): set up 2 sockets
Sep 26 04:02:45 dist-test-slave-nhpw krb5kdc[22964](info): commencing operation
krb5kdc: starting...
W20260926 04:02:46.352891 392 mini_kdc.cc:121] Time spent starting KDC: real 1.064s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395095474643-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395095474643-392-0/krb5kdc/test-user.keytab.
Sep 26 04:02:46 dist-test-slave-nhpw krb5kdc[22964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395366, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T04:02:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T04:02:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260926 04:02:46.463984 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395095474643-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395095474643-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395095474643-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43783
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43783,127.0.98.61:39949,127.0.98.60:34045
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 04:02:46.543280 22977 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 04:02:46.543479 22977 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:02:46.543496 22977 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:02:46.543515 22977 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:02:46.544679 22977 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:02:46.544718 22977 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:02:46.544731 22977 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:02:46.544744 22977 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:02:46.544782 22977 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:02:46.546252 22977 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395095474643-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395095474643-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43783,127.0.98.61:39949,127.0.98.60:34045
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395095474643-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22977
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395095474643-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:02:46.546492 22977 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:02:46.546717 22977 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:02:46.551434 22983 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:02:46.551445 22982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:02:46.551704 22977 server_base.cc:1034] running on GCE node
W20260926 04:02:46.551445 22985 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:02:46.552008 22977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:02:46.552295 22977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:02:46.553457 22977 hybrid_clock.cc:648] HybridClock initialized: now 1790395366553445 us; error 28 us; skew 500 ppm
Sep 26 04:02:46 dist-test-slave-nhpw krb5kdc[22964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395366, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:02:46.555773 22977 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260926 04:02:46.556170 22977 webserver.cc:460] Webserver started at http://127.0.98.62:37779/ using document root <none> and password file <none>
I20260926 04:02:46.556387 22977 fs_manager.cc:359] Metadata directory not provided
I20260926 04:02:46.556452 22977 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:02:46.556598 22977 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:02:46.557605 22977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395095474643-392-0/minicluster-data/master-0/data/instance:
uuid: "8382c3742b944661b02a6f3bae8b3ffc"
format_stamp: "Formatted at 2026-09-26 04:02:46 on dist-test-slave-nhpw"
server_key: "61e31260fe25d3b7eecabd032046ea51"
server_key_iv: "edfea75b15dd61158e157bad1ae789bc"
server_key_version: "encryptionkey@0"
I20260926 04:02:46.557965 22977 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395095474643-392-0/minicluster-data/master-0/wal/instance:
uuid: "8382c3742b944661b02a6f3bae8b3ffc"
format_stamp: "Formatted at 2026-09-26 04:02:46 on dist-test-slave-nhpw"
server_key: "61e31260fe25d3b7eecabd032046ea51"
server_key_iv: "edfea75b15dd61158e157bad1ae789bc"
server_key_version: "encryptionkey@0"
I20260926 04:02:46.559258 22977 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 04:02:46.559957 22992 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:46.560118 22977 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 04:02:46.560218 22977 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395095474643-392-0/minicluster-data/master-0/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395095474643-392-0/minicluster-data/master-0/wal
uuid: "8382c3742b944661b02a6f3bae8b3ffc"
format_stamp: "Formatted at 2026-09-26 04:02:46 on dist-test-slave-nhpw"
server_key: "61e31260fe25d3b7eecabd032046ea51"
server_key_iv: "edfea75b15dd61158e157bad1ae789bc"
server_key_version: "encryptionkey@0"
I20260926 04:02:46.560299 22977 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395095474643-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395095474643-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395095474643-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:02:46.594048 22977 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:02:46.595903 22977 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:02:46.596060 22977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:02:46.599958 22977 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43783
I20260926 04:02:46.600010 23044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43783 every 8 connection(s)
I20260926 04:02:46.600358 22977 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395095474643-392-0/minicluster-data/master-0/data/info.pb
I20260926 04:02:46.601346 23045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:02:46.602805 23045 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43783 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 04:02:46 dist-test-slave-nhpw krb5kdc[22964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395366, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 04:02:46.608358 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 22977
I20260926 04:02:46.608486 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395095474643-392-0/minicluster-data/master-0/wal/instance
I20260926 04:02:46.608717 392 external_mini_cluster.cc:1428] Setting key 4bc9384ad40ff99dc4e097290a6cc07b
I20260926 04:02:46.612084 23048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:46.603496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37308 (local address 127.0.98.62:43783)
0926 04:02:46.603657 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:46.603666 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:46.603818 (+ 152us) server_negotiation.cc:407] Connection header received
0926 04:02:46.604319 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:46.604320 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:46.604336 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:46.604412 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:46.604968 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:46.605757 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:46.606564 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:46.606856 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:46.606933 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:46.608932 (+ 1999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:46.608940 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:46.608940 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:02:46.608948 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:46.610586 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:46.611044 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:46.611047 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:46.611047 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:46.611117 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:46.611365 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:46.611365 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:46.611366 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:46.611545 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:46.611603 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:46.611827 (+ 224us) server_negotiation.cc:299] Negotiation successful
0926 04:02:46.611942 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":122,"thread_start_us":90,"threads_started":1}
I20260926 04:02:46.612205 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:46.603477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43783 (local address 127.0.0.1:37308)
0926 04:02:46.603689 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:46.603733 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 04:02:46.604195 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:02:46.604437 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:02:46.604440 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:02:46.604736 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:02:46.604884 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:46.604887 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:46.605837 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:46.605838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:46.606491 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:46.606492 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:46.606943 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:46.606944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:46.606998 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:46.607385 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:02:46.607393 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:02:46.608845 (+ 1452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:02:46.610678 (+ 1833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:02:46.610684 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:02:46.610686 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:02:46.610961 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:02:46.611194 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:02:46.611194 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:02:46.611195 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:02:46.611301 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:02:46.611612 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:02:46.611618 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:02:46.611707 (+ 89us) client_negotiation.cc:769] Sending connection context
0926 04:02:46.611812 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 04:02:46.612027 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":63,"thread_start_us":85,"threads_started":1}
I20260926 04:02:46.613169 23045 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39949 } has no permanent_uuid. Determining permanent_uuid...
I20260926 04:02:46.613515 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:46.613378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39949 (local address 127.0.0.1:50056)
0926 04:02:46.613431 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:46.613484 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 04:02:46.613665 22996 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39949: connect: Connection refused (error 111)
I20260926 04:02:46.614090 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:46.613972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39949 (local address 127.0.0.1:50068)
0926 04:02:46.614028 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:46.614060 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 04:02:46.614252 23045 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:39949: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39949: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395095474643-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395095474643-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260926 04:02:46.631484 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395095474643-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395095474643-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395095474643-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395095474643-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:39949
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395095474643-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43783,127.0.98.61:39949,127.0.98.60:34045
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 04:02:46.663470 23045 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39949 } attempt: 1
I20260926 04:02:46.664000 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:46.663857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39949 (local address 127.0.0.1:50074)
0926 04:02:46.663914 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:46.663963 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260926 04:02:46.664355 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:46.664249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39949 (local address 127.0.0.1:50078)
0926 04:02:46.664304 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:46.664334 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 04:02:46.664440 23045 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:39949: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39949: connect: Connection refused (error 111)
W20260926 04:02:46.726358 23052 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 04:02:46.726557 23052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:02:46.726573 23052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:02:46.726593 23052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:02:46.727624 23052 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:02:46.727658 23052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:02:46.727672 23052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:02:46.727684 23052 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:02:46.727696 23052 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:02:46.729151 23052 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395095474643-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395095474643-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43783,127.0.98.61:39949,127.0.98.60:34045
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395095474643-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:39949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395095474643-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23052
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395095474643-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:02:46.729329 23052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:02:46.729483 23052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 04:02:46.732594 23045 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39949 } attempt: 2
I20260926 04:02:46.733105 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:46.732980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39949 (local address 127.0.0.1:50094)
0926 04:02:46.733035 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:46.733069 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 04:02:46.733489 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:46.733370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39949 (local address 127.0.0.1:50106)
0926 04:02:46.733428 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:46.733459 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 04:02:46.733574 23045 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:39949: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39949: connect: Connection refused (error 111)
I20260926 04:02:46.734200 23052 server_base.cc:1034] running on GCE node
W20260926 04:02:46.734169 23057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:02:46.734167 23060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:02:46.734136 23058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:02:46.734524 23052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:02:46.734812 23052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:02:46.735948 23052 hybrid_clock.cc:648] HybridClock initialized: now 1790395366735936 us; error 25 us; skew 500 ppm
Sep 26 04:02:46 dist-test-slave-nhpw krb5kdc[22964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395366, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:02:46.738406 23052 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260926 04:02:46.738847 23052 webserver.cc:460] Webserver started at http://127.0.98.61:42225/ using document root <none> and password file <none>
I20260926 04:02:46.739066 23052 fs_manager.cc:359] Metadata directory not provided
I20260926 04:02:46.739132 23052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:02:46.739285 23052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:02:46.740305 23052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395095474643-392-0/minicluster-data/master-1/data/instance:
uuid: "81b6de3f7bcd4d1392983f510cfac45d"
format_stamp: "Formatted at 2026-09-26 04:02:46 on dist-test-slave-nhpw"
server_key: "682a39be26fffc65eecca71bfeeb6e03"
server_key_iv: "e799923cf8265f5bdb74cb6e26d9ebbb"
server_key_version: "encryptionkey@0"
I20260926 04:02:46.740726 23052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395095474643-392-0/minicluster-data/master-1/wal/instance:
uuid: "81b6de3f7bcd4d1392983f510cfac45d"
format_stamp: "Formatted at 2026-09-26 04:02:46 on dist-test-slave-nhpw"
server_key: "682a39be26fffc65eecca71bfeeb6e03"
server_key_iv: "e799923cf8265f5bdb74cb6e26d9ebbb"
server_key_version: "encryptionkey@0"
I20260926 04:02:46.742048 23052 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 04:02:46.742837 23067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:46.743037 23052 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 04:02:46.743140 23052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395095474643-392-0/minicluster-data/master-1/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395095474643-392-0/minicluster-data/master-1/wal
uuid: "81b6de3f7bcd4d1392983f510cfac45d"
format_stamp: "Formatted at 2026-09-26 04:02:46 on dist-test-slave-nhpw"
server_key: "682a39be26fffc65eecca71bfeeb6e03"
server_key_iv: "e799923cf8265f5bdb74cb6e26d9ebbb"
server_key_version: "encryptionkey@0"
I20260926 04:02:46.743227 23052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395095474643-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395095474643-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395095474643-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:02:46.770539 23052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:02:46.772576 23052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:02:46.772776 23052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:02:46.776718 23052 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:39949
I20260926 04:02:46.776769 23119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:39949 every 8 connection(s)
I20260926 04:02:46.777139 23052 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395095474643-392-0/minicluster-data/master-1/data/info.pb
I20260926 04:02:46.778136 23120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:02:46.779667 23120 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43783 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 04:02:46 dist-test-slave-nhpw krb5kdc[22964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395366, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 04:02:46.785920 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 23052
I20260926 04:02:46.786044 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395095474643-392-0/minicluster-data/master-1/wal/instance
I20260926 04:02:46.786254 392 external_mini_cluster.cc:1428] Setting key 420013940cd5d64fc4e68d31d4c14429
I20260926 04:02:46.789582 23048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:46.780407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37322 (local address 127.0.98.62:43783)
0926 04:02:46.780468 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:46.780470 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:46.780785 (+ 315us) server_negotiation.cc:407] Connection header received
0926 04:02:46.781292 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:46.781293 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:46.781309 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:46.781376 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:46.782080 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:46.782601 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:46.783659 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:46.783891 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:46.783958 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:46.786361 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:46.786362 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:46.786363 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:02:46.786370 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:46.787868 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:46.788372 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:46.788373 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:46.788374 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:46.788434 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:46.788736 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:46.788737 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:46.788738 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:46.788884 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:46.788958 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:46.789344 (+ 386us) server_negotiation.cc:299] Negotiation successful
0926 04:02:46.789442 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 04:02:46.789744 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:46.780369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43783 (local address 127.0.0.1:37322)
0926 04:02:46.780606 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:46.780677 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0926 04:02:46.781193 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:02:46.781459 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:02:46.781462 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:02:46.781739 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:02:46.781947 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:46.781952 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:46.782675 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:46.782676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:46.783550 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:46.783553 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:46.783982 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:46.783983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:46.784087 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:46.784694 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:02:46.784704 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:02:46.786240 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:02:46.787953 (+ 1713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:02:46.787960 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:02:46.787962 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:02:46.788287 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:02:46.788530 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:02:46.788531 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:02:46.788531 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:02:46.788678 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:02:46.789009 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:02:46.789016 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:02:46.789150 (+ 134us) client_negotiation.cc:769] Sending connection context
0926 04:02:46.789285 (+ 135us) client_negotiation.cc:240] Negotiation successful
0926 04:02:46.789437 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":162,"threads_started":1}
I20260926 04:02:46.790373 23120 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39949 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 26 04:02:46 dist-test-slave-nhpw krb5kdc[22964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395366, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 04:02:46.800784 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:46.790659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39949 (local address 127.0.0.1:50122)
0926 04:02:46.790894 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:46.790906 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 04:02:46.790964 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:02:46.791105 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:02:46.791106 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:02:46.791265 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:02:46.791328 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:46.791330 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:46.792187 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:46.792188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:46.793099 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:46.793101 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:46.793582 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:46.793583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:46.793672 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:46.794105 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:02:46.794110 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:02:46.797618 (+ 3508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:02:46.799679 (+ 2061us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:02:46.799680 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:02:46.799681 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:02:46.799846 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:02:46.800119 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:02:46.800120 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:02:46.800120 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:02:46.800181 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:02:46.800463 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:02:46.800464 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:02:46.800502 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 04:02:46.800577 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 04:02:46.800708 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218}
I20260926 04:02:46.800936 23124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:46.790741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50122 (local address 127.0.98.61:39949)
0926 04:02:46.790885 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:46.790888 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:46.790956 (+ 68us) server_negotiation.cc:407] Connection header received
0926 04:02:46.791024 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:46.791025 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:46.791037 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:46.791101 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:46.791432 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:46.792104 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:46.793198 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:46.793499 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:46.793570 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:46.797723 (+ 4153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:46.797730 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:46.797731 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:02:46.797736 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:46.799598 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:46.799972 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:46.799976 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:46.799976 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:46.800036 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:46.800245 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:46.800246 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:46.800246 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:46.800390 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:46.800483 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:46.800687 (+ 204us) server_negotiation.cc:299] Negotiation successful
0926 04:02:46.800785 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":94,"threads_started":1}
I20260926 04:02:46.801550 23120 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34045 } has no permanent_uuid. Determining permanent_uuid...
I20260926 04:02:46.801923 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:46.801783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34045 (local address 127.0.0.1:59822)
0926 04:02:46.801838 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:46.801892 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 04:02:46.802083 23068 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34045: connect: Connection refused (error 111)
I20260926 04:02:46.802526 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:46.802431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34045 (local address 127.0.0.1:59826)
0926 04:02:46.802485 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:46.802510 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 04:02:46.802644 23120 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34045: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34045: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395095474643-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395095474643-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260926 04:02:46.813868 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task7pf3XC/build/release/bin/kudu
/tmp/dist-test-task7pf3XC/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395095474643-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395095474643-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395095474643-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395095474643-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:34045
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395095474643-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37525
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43783,127.0.98.61:39949,127.0.98.60:34045
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395095474643-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395095474643-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395095474643-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 04:02:46.824719 23045 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39949 } attempt: 3
Sep 26 04:02:46 dist-test-slave-nhpw krb5kdc[22964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395366, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 04:02:46.832024 23124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:46.825169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50136 (local address 127.0.98.61:39949)
0926 04:02:46.825232 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:46.825234 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:46.825245 (+ 11us) server_negotiation.cc:407] Connection header received
0926 04:02:46.825309 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:46.825310 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:46.825322 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:46.825398 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:46.825723 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:46.826213 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:46.826939 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:46.827157 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:46.827246 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:46.829434 (+ 2188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:46.829435 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:46.829435 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:02:46.829443 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:46.830879 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:46.831291 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:46.831292 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:46.831292 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:46.831351 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:46.831560 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:46.831561 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:46.831561 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:46.831675 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:46.831726 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:46.831887 (+ 161us) server_negotiation.cc:299] Negotiation successful
0926 04:02:46.831956 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 04:02:46.832087 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:46.825096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39949 (local address 127.0.0.1:50136)
0926 04:02:46.825169 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:46.825179 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 04:02:46.825239 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:02:46.825402 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:02:46.825403 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:02:46.825564 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:02:46.825631 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:46.825632 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:46.826308 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:46.826309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:46.826839 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:46.826840 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:46.827245 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:46.827246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:46.827335 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:46.827897 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:02:46.827901 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:02:46.829331 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:02:46.830962 (+ 1631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:02:46.830963 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:02:46.830964 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:02:46.831197 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:02:46.831428 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:02:46.831429 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:02:46.831429 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:02:46.831498 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:02:46.831738 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:02:46.831739 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:02:46.831792 (+ 53us) client_negotiation.cc:769] Sending connection context
0926 04:02:46.831862 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 04:02:46.831963 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 04:02:46.832568 23045 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34045 } has no permanent_uuid. Determining permanent_uuid...
I20260926 04:02:46.832952 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:46.832832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34045 (local address 127.0.0.1:59842)
0926 04:02:46.832891 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:46.832927 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 04:02:46.833280 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:46.833189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34045 (local address 127.0.0.1:59844)
0926 04:02:46.833236 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:46.833263 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 04:02:46.833377 23045 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34045: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34045: connect: Connection refused (error 111)
I20260926 04:02:46.851913 23120 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34045 } attempt: 1
I20260926 04:02:46.852526 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:46.852343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34045 (local address 127.0.0.1:59848)
0926 04:02:46.852424 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:46.852481 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260926 04:02:46.853010 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:46.852845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34045 (local address 127.0.0.1:59850)
0926 04:02:46.852905 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:46.852985 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260926 04:02:46.853148 23120 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34045: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34045: connect: Connection refused (error 111)
I20260926 04:02:46.864538 23045 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34045 } attempt: 1
I20260926 04:02:46.865041 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:46.864897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34045 (local address 127.0.0.1:59866)
0926 04:02:46.864957 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:46.865009 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 04:02:46.865404 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:46.865312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34045 (local address 127.0.0.1:59872)
0926 04:02:46.865355 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:46.865385 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 04:02:46.865592 23045 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34045: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34045: connect: Connection refused (error 111)
W20260926 04:02:46.899039 23127 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 04:02:46.899247 23127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 04:02:46.899264 23127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 04:02:46.899284 23127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 04:02:46.900439 23127 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 04:02:46.900480 23127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 04:02:46.900502 23127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 04:02:46.900514 23127 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 04:02:46.900527 23127 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 04:02:46.902016 23127 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37525
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395095474643-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395095474643-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43783,127.0.98.61:39949,127.0.98.60:34045
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395095474643-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:34045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395095474643-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23127
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395095474643-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 02:52:23 UTC on ccycloud.quasar-vncnvg.root.comops.site
build id 4673638
I20260926 04:02:46.902258 23127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 04:02:46.902510 23127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 04:02:46.907266 23135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:02:46.907349 23127 server_base.cc:1034] running on GCE node
W20260926 04:02:46.907245 23132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 04:02:46.907239 23133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 04:02:46.907724 23127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 04:02:46.908030 23127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 04:02:46.909219 23127 hybrid_clock.cc:648] HybridClock initialized: now 1790395366909206 us; error 44 us; skew 500 ppm
Sep 26 04:02:46 dist-test-slave-nhpw krb5kdc[22964](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395366, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 04:02:46.911542 23127 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260926 04:02:46.912000 23127 webserver.cc:460] Webserver started at http://127.0.98.60:34687/ using document root <none> and password file <none>
I20260926 04:02:46.912225 23127 fs_manager.cc:359] Metadata directory not provided
I20260926 04:02:46.912289 23127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 04:02:46.912441 23127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 04:02:46.913575 23127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395095474643-392-0/minicluster-data/master-2/data/instance:
uuid: "c265276c53f34cf091117951d3f48e8d"
format_stamp: "Formatted at 2026-09-26 04:02:46 on dist-test-slave-nhpw"
server_key: "373f336f14eb52c30a41614f485ce35a"
server_key_iv: "015d0993603952de6267c171b287bad1"
server_key_version: "encryptionkey@0"
I20260926 04:02:46.913933 23127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395095474643-392-0/minicluster-data/master-2/wal/instance:
uuid: "c265276c53f34cf091117951d3f48e8d"
format_stamp: "Formatted at 2026-09-26 04:02:46 on dist-test-slave-nhpw"
server_key: "373f336f14eb52c30a41614f485ce35a"
server_key_iv: "015d0993603952de6267c171b287bad1"
server_key_version: "encryptionkey@0"
I20260926 04:02:46.915203 23127 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 04:02:46.915963 23142 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 04:02:46.916147 23127 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 04:02:46.916241 23127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395095474643-392-0/minicluster-data/master-2/data,/tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395095474643-392-0/minicluster-data/master-2/wal
uuid: "c265276c53f34cf091117951d3f48e8d"
format_stamp: "Formatted at 2026-09-26 04:02:46 on dist-test-slave-nhpw"
server_key: "373f336f14eb52c30a41614f485ce35a"
server_key_iv: "015d0993603952de6267c171b287bad1"
server_key_version: "encryptionkey@0"
I20260926 04:02:46.916325 23127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395095474643-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395095474643-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395095474643-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 04:02:46.921317 23120 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34045 } attempt: 2
I20260926 04:02:46.921742 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:46.921607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34045 (local address 127.0.0.1:59880)
0926 04:02:46.921668 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:46.921708 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 04:02:46.922091 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:46.921999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34045 (local address 127.0.0.1:59882)
0926 04:02:46.922044 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:46.922070 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 04:02:46.922201 23120 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34045: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34045: connect: Connection refused (error 111)
I20260926 04:02:46.940675 23045 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34045 } attempt: 2
I20260926 04:02:46.941133 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:46.941014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34045 (local address 127.0.0.1:59892)
0926 04:02:46.941070 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:46.941099 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 04:02:46.941447 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:46.941337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34045 (local address 127.0.0.1:59906)
0926 04:02:46.941405 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:46.941426 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260926 04:02:46.941519 23045 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:34045: Network error: Client connection negotiation failed: client connection to 127.0.98.60:34045: connect: Connection refused (error 111)
I20260926 04:02:46.941797 23127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 04:02:46.944384 23127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 04:02:46.944510 23127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 04:02:46.949148 23127 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:34045
I20260926 04:02:46.949179 23194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:34045 every 8 connection(s)
I20260926 04:02:46.949532 23127 server_base.cc:1166] Dumped server information to /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395095474643-392-0/minicluster-data/master-2/data/info.pb
I20260926 04:02:46.950495 23195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 04:02:46.952075 23195 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43783 } has no permanent_uuid. Determining permanent_uuid...
I20260926 04:02:46.955015 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task7pf3XC/build/release/bin/kudu as pid 23127
I20260926 04:02:46.955128 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task7pf3XC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790395095474643-392-0/minicluster-data/master-2/wal/instance
I20260926 04:02:46.955332 392 external_mini_cluster.cc:1428] Setting key 1d1519453ec178e9206b4b656276c970
Sep 26 04:02:46 dist-test-slave-nhpw krb5kdc[22964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395366, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 26 04:02:46 dist-test-slave-nhpw krb5kdc[22964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395366, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 04:02:46.962597 23048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:46.952871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37330 (local address 127.0.98.62:43783)
0926 04:02:46.952959 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:46.952962 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:46.953125 (+ 163us) server_negotiation.cc:407] Connection header received
0926 04:02:46.953631 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:46.953632 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:46.953642 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:46.953705 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:46.954315 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:46.954782 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:46.956253 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:46.956574 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:46.956682 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:46.959210 (+ 2528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:46.959211 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:46.959211 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:02:46.959218 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:46.960925 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:46.961442 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:46.961443 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:46.961444 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:46.961510 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:46.961793 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:46.961795 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:46.961795 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:46.961969 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:46.962060 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:46.962400 (+ 340us) server_negotiation.cc:299] Negotiation successful
0926 04:02:46.962471 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 04:02:46.962850 23197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:46.952823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43783 (local address 127.0.0.1:37330)
0926 04:02:46.952988 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:46.953037 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 04:02:46.953476 (+ 439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:02:46.953775 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:02:46.953779 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:02:46.954034 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:02:46.954219 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:46.954223 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:46.954893 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:46.954895 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:46.956174 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:46.956176 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:46.956745 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:46.956746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:46.956874 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:46.957370 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:02:46.957380 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:02:46.959053 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:02:46.961064 (+ 2011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:02:46.961071 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:02:46.961073 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:02:46.961372 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:02:46.961590 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:02:46.961591 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:02:46.961592 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:02:46.961716 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:02:46.962067 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:02:46.962074 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:02:46.962243 (+ 169us) client_negotiation.cc:769] Sending connection context
0926 04:02:46.962370 (+ 127us) client_negotiation.cc:240] Negotiation successful
0926 04:02:46.962533 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":94,"threads_started":1}
I20260926 04:02:46.963579 23195 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39949 } has no permanent_uuid. Determining permanent_uuid...
I20260926 04:02:46.965178 23199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:46.955730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59910 (local address 127.0.98.60:34045)
0926 04:02:46.955904 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:46.955908 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:46.955925 (+ 17us) server_negotiation.cc:407] Connection header received
0926 04:02:46.955972 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:46.955974 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:46.955992 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:46.956072 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:46.956848 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:46.957899 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:46.958976 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:46.959309 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:46.959398 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:46.961685 (+ 2287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:46.961692 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:46.961693 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:02:46.961699 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:46.963670 (+ 1971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:46.964286 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:46.964288 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:46.964289 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:46.964346 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:46.964558 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:46.964559 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:46.964559 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:46.964769 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:46.964933 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:46.964979 (+ 46us) server_negotiation.cc:299] Negotiation successful
0926 04:02:46.965070 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":96,"threads_started":1}
Sep 26 04:02:46 dist-test-slave-nhpw krb5kdc[22964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395366, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 26 04:02:46 dist-test-slave-nhpw krb5kdc[22964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395366, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260926 04:02:46.971203 23197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:46.963860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39949 (local address 127.0.0.1:50138)
0926 04:02:46.963909 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:46.963919 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 04:02:46.963954 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:02:46.964146 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:02:46.964147 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:02:46.964295 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:02:46.964334 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:46.964335 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:46.965351 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:46.965352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:46.966215 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:46.966217 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:46.966633 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:46.966634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:46.966681 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:46.967057 (+ 376us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:02:46.967061 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:02:46.968223 (+ 1162us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:02:46.970096 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:02:46.970097 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:02:46.970098 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:02:46.970317 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:02:46.970564 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:02:46.970565 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:02:46.970566 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:02:46.970640 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:02:46.970915 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:02:46.970916 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:02:46.970986 (+ 70us) client_negotiation.cc:769] Sending connection context
0926 04:02:46.971067 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 04:02:46.971139 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 04:02:46.971338 23124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:46.963969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50138 (local address 127.0.98.61:39949)
0926 04:02:46.964012 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:46.964014 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:46.964028 (+ 14us) server_negotiation.cc:407] Connection header received
0926 04:02:46.964061 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:46.964062 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:46.964071 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:46.964123 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:46.964443 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:46.965266 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:46.966285 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:46.966559 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:46.966638 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:46.968320 (+ 1682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:46.968322 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:46.968322 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:02:46.968331 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:46.969988 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:46.970410 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:46.970412 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:46.970413 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:46.970482 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:46.970720 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:46.970720 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:46.970721 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:46.970824 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:46.970878 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:46.971124 (+ 246us) server_negotiation.cc:299] Negotiation successful
0926 04:02:46.971211 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 04:02:46.971747 23195 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34045 } has no permanent_uuid. Determining permanent_uuid...
I20260926 04:02:46.973964 23048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:46.966006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37334 (local address 127.0.98.62:43783)
0926 04:02:46.966042 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:46.966044 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:46.966053 (+ 9us) server_negotiation.cc:407] Connection header received
0926 04:02:46.966099 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:46.966100 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:46.966109 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:46.966178 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:46.966577 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:46.967300 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:46.968013 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:46.968193 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:46.968266 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:46.970957 (+ 2691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:46.970958 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:46.970958 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:02:46.970963 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:46.972784 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:46.973158 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:46.973159 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:46.973159 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:46.973224 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:46.973440 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:46.973442 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:46.973442 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:46.973563 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:46.973746 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:46.973786 (+ 40us) server_negotiation.cc:299] Negotiation successful
0926 04:02:46.973859 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
Sep 26 04:02:46 dist-test-slave-nhpw krb5kdc[22964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395366, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 26 04:02:46 dist-test-slave-nhpw krb5kdc[22964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395366, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260926 04:02:46.979516 23197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:46.972019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34045 (local address 127.0.0.1:59920)
0926 04:02:46.972061 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:46.972070 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 04:02:46.972103 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:02:46.972860 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:02:46.972861 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:02:46.972992 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:02:46.973016 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:46.973017 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:46.973634 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:46.973635 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:46.974250 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:46.974252 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:46.974764 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:46.974765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:46.974833 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:46.975276 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:02:46.975282 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:02:46.976429 (+ 1147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:02:46.978406 (+ 1977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:02:46.978407 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:02:46.978407 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:02:46.978577 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:02:46.978805 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:02:46.978805 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:02:46.978806 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:02:46.978876 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:02:46.979166 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:02:46.979166 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:02:46.979202 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 04:02:46.979286 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 04:02:46.979394 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260926 04:02:46.979594 23199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:46.972100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59920 (local address 127.0.98.60:34045)
0926 04:02:46.972736 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:46.972737 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:46.972750 (+ 13us) server_negotiation.cc:407] Connection header received
0926 04:02:46.972775 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:46.972775 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:46.972784 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:46.972859 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:46.973091 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:46.973538 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:46.974347 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:46.974655 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:46.974729 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:46.976517 (+ 1788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:46.976519 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:46.976519 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:02:46.976527 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:46.978296 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:46.978660 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:46.978661 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:46.978662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:46.978724 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:46.978946 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:46.978947 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:46.978948 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:46.979088 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:46.979169 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:46.979313 (+ 144us) server_negotiation.cc:299] Negotiation successful
0926 04:02:46.979466 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":84,"server-negotiator.queue_time_us":615}
I20260926 04:02:46.981148 23195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c265276c53f34cf091117951d3f48e8d: Bootstrap starting.
I20260926 04:02:46.982036 23195 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c265276c53f34cf091117951d3f48e8d: Neither blocks nor log segments found. Creating new log.
I20260926 04:02:46.982239 23124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:46.974836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50152 (local address 127.0.98.61:39949)
0926 04:02:46.974879 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:46.974881 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:46.974888 (+ 7us) server_negotiation.cc:407] Connection header received
0926 04:02:46.974907 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:46.974908 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:46.974916 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:46.974970 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:46.975418 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:46.975857 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:46.976557 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:46.976835 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:46.976917 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:46.979239 (+ 2322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:46.979240 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:46.979241 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:02:46.979247 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:46.980997 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:46.981424 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:46.981425 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:46.981426 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:46.981484 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:46.981689 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:46.981691 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:46.981691 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:46.981801 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:46.981967 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:46.982004 (+ 37us) server_negotiation.cc:299] Negotiation successful
0926 04:02:46.982119 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 04:02:46.982365 23195 log.cc:824] T 00000000000000000000000000000000 P c265276c53f34cf091117951d3f48e8d: Log is configured to *not* fsync() on all Append() calls
I20260926 04:02:46.983242 23195 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c265276c53f34cf091117951d3f48e8d: No bootstrap required, opened a new log
I20260926 04:02:46.983742 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 04:02:46.985276 23195 raft_consensus.cc:348] T 00000000000000000000000000000000 P c265276c53f34cf091117951d3f48e8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8382c3742b944661b02a6f3bae8b3ffc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43783 } } peers { permanent_uuid: "81b6de3f7bcd4d1392983f510cfac45d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39949 } } peers { permanent_uuid: "c265276c53f34cf091117951d3f48e8d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34045 } }
I20260926 04:02:46.985407 23195 raft_consensus.cc:729] T 00000000000000000000000000000000 P c265276c53f34cf091117951d3f48e8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c265276c53f34cf091117951d3f48e8d, State: Initialized, Role: FOLLOWER
I20260926 04:02:46.984808 23019 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "8382c3742b944661b02a6f3bae8b3ffc"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37334
I20260926 04:02:46.985498 23195 consensus_queue.cc:260] T 00000000000000000000000000000000 P c265276c53f34cf091117951d3f48e8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8382c3742b944661b02a6f3bae8b3ffc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43783 } } peers { permanent_uuid: "81b6de3f7bcd4d1392983f510cfac45d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39949 } } peers { permanent_uuid: "c265276c53f34cf091117951d3f48e8d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34045 } }
I20260926 04:02:46.985819 23195 sys_catalog.cc:564] T 00000000000000000000000000000000 P c265276c53f34cf091117951d3f48e8d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:02:46.985797 23201 sys_catalog.cc:455] T 00000000000000000000000000000000 P c265276c53f34cf091117951d3f48e8d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8382c3742b944661b02a6f3bae8b3ffc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43783 } } peers { permanent_uuid: "81b6de3f7bcd4d1392983f510cfac45d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39949 } } peers { permanent_uuid: "c265276c53f34cf091117951d3f48e8d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34045 } } }
I20260926 04:02:46.985896 23201 sys_catalog.cc:458] T 00000000000000000000000000000000 P c265276c53f34cf091117951d3f48e8d [sys.catalog]: This master's current role is: FOLLOWER
W20260926 04:02:46.987576 23212 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c265276c53f34cf091117951d3f48e8d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 04:02:46.987632 23212 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 04:02:46.987989 23127 master_runner.cc:186] Error getting master registration for 127.0.98.62:43783: OK, instance_id { permanent_uuid: "8382c3742b944661b02a6f3bae8b3ffc" instance_seqno: 1790395366598907 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 04:02:46.990356 23127 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 04:02:47.013368 23120 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34045 } attempt: 3
Sep 26 04:02:47 dist-test-slave-nhpw krb5kdc[22964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395366, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 04:02:47.020732 23122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:47.013815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34045 (local address 127.0.0.1:59926)
0926 04:02:47.013893 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:47.013908 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 04:02:47.013981 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:02:47.014152 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:02:47.014153 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:02:47.014317 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:02:47.014371 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:47.014373 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:47.015026 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:47.015027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:47.015526 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:47.015528 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:47.015897 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:47.015899 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:47.015962 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:47.016427 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:02:47.016432 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:02:47.017712 (+ 1280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:02:47.019555 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:02:47.019556 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:02:47.019557 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:02:47.019775 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:02:47.019987 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:02:47.019988 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:02:47.019988 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:02:47.020053 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:02:47.020375 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:02:47.020375 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:02:47.020423 (+ 48us) client_negotiation.cc:769] Sending connection context
0926 04:02:47.020494 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 04:02:47.020599 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260926 04:02:47.020741 23199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:47.013927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59926 (local address 127.0.98.60:34045)
0926 04:02:47.013979 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:47.013981 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:47.013992 (+ 11us) server_negotiation.cc:407] Connection header received
0926 04:02:47.014047 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:47.014048 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:47.014059 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:47.014131 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:47.014454 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:47.014930 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:47.015614 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:47.015812 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:47.015888 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:47.017798 (+ 1910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:47.017800 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:47.017800 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:02:47.017808 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:47.019467 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:47.019852 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:47.019853 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:47.019854 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:47.019917 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:47.020122 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:47.020123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:47.020123 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:47.020303 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:47.020377 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:47.020534 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 04:02:47.020617 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 04:02:47.022739 23120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81b6de3f7bcd4d1392983f510cfac45d: Bootstrap starting.
I20260926 04:02:47.023489 23120 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81b6de3f7bcd4d1392983f510cfac45d: Neither blocks nor log segments found. Creating new log.
I20260926 04:02:47.023810 23120 log.cc:824] T 00000000000000000000000000000000 P 81b6de3f7bcd4d1392983f510cfac45d: Log is configured to *not* fsync() on all Append() calls
I20260926 04:02:47.024571 23120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81b6de3f7bcd4d1392983f510cfac45d: No bootstrap required, opened a new log
I20260926 04:02:47.025987 23120 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81b6de3f7bcd4d1392983f510cfac45d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8382c3742b944661b02a6f3bae8b3ffc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43783 } } peers { permanent_uuid: "81b6de3f7bcd4d1392983f510cfac45d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39949 } } peers { permanent_uuid: "c265276c53f34cf091117951d3f48e8d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34045 } }
I20260926 04:02:47.026129 23120 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81b6de3f7bcd4d1392983f510cfac45d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81b6de3f7bcd4d1392983f510cfac45d, State: Initialized, Role: FOLLOWER
I20260926 04:02:47.026218 23120 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81b6de3f7bcd4d1392983f510cfac45d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8382c3742b944661b02a6f3bae8b3ffc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43783 } } peers { permanent_uuid: "81b6de3f7bcd4d1392983f510cfac45d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39949 } } peers { permanent_uuid: "c265276c53f34cf091117951d3f48e8d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34045 } }
I20260926 04:02:47.026501 23215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81b6de3f7bcd4d1392983f510cfac45d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8382c3742b944661b02a6f3bae8b3ffc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43783 } } peers { permanent_uuid: "81b6de3f7bcd4d1392983f510cfac45d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39949 } } peers { permanent_uuid: "c265276c53f34cf091117951d3f48e8d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34045 } } }
I20260926 04:02:47.026585 23120 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81b6de3f7bcd4d1392983f510cfac45d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:02:47.026631 23215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81b6de3f7bcd4d1392983f510cfac45d [sys.catalog]: This master's current role is: FOLLOWER
W20260926 04:02:47.028378 23226 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 81b6de3f7bcd4d1392983f510cfac45d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 04:02:47.028461 23226 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 04:02:47.028543 23052 master_runner.cc:186] Error getting master registration for 127.0.98.62:43783: OK, instance_id { permanent_uuid: "8382c3742b944661b02a6f3bae8b3ffc" instance_seqno: 1790395366598907 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 04:02:47.031490 23052 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 04:02:47.040659 23045 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34045 } attempt: 3
Sep 26 04:02:47 dist-test-slave-nhpw krb5kdc[22964](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790395366, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260926 04:02:47.049438 23199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:47.041108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59938 (local address 127.0.98.60:34045)
0926 04:02:47.041181 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:47.041182 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:47.041208 (+ 26us) server_negotiation.cc:407] Connection header received
0926 04:02:47.041236 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:47.041237 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:47.041248 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:47.041328 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:47.041704 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:47.042478 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:47.043503 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:47.043813 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:47.043895 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:47.046239 (+ 2344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:47.046241 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:47.046241 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:02:47.046249 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:47.048062 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:47.048503 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:47.048504 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:47.048504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:47.048590 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:47.048867 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:47.048868 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:47.048869 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:47.049013 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:47.049083 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:47.049259 (+ 176us) server_negotiation.cc:299] Negotiation successful
0926 04:02:47.049340 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 04:02:47.049439 23047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:47.041009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:34045 (local address 127.0.0.1:59938)
0926 04:02:47.041067 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 04:02:47.041078 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 04:02:47.041120 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 04:02:47.041348 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 04:02:47.041349 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 04:02:47.041542 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0926 04:02:47.041600 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:47.041602 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:47.042559 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:47.042560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:47.043398 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 04:02:47.043401 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 04:02:47.043899 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 04:02:47.043900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 04:02:47.043960 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:47.044482 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 04:02:47.044487 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 04:02:47.046154 (+ 1667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 04:02:47.048167 (+ 2013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:02:47.048168 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:02:47.048168 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:02:47.048403 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:02:47.048708 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 04:02:47.048709 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 04:02:47.048709 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 04:02:47.048781 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 04:02:47.049082 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 04:02:47.049084 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 04:02:47.049142 (+ 58us) client_negotiation.cc:769] Sending connection context
0926 04:02:47.049220 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 04:02:47.049330 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 04:02:47.051585 23045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8382c3742b944661b02a6f3bae8b3ffc: Bootstrap starting.
I20260926 04:02:47.052346 23045 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8382c3742b944661b02a6f3bae8b3ffc: Neither blocks nor log segments found. Creating new log.
I20260926 04:02:47.052663 23045 log.cc:824] T 00000000000000000000000000000000 P 8382c3742b944661b02a6f3bae8b3ffc: Log is configured to *not* fsync() on all Append() calls
I20260926 04:02:47.053515 23045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8382c3742b944661b02a6f3bae8b3ffc: No bootstrap required, opened a new log
I20260926 04:02:47.054891 23045 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8382c3742b944661b02a6f3bae8b3ffc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8382c3742b944661b02a6f3bae8b3ffc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43783 } } peers { permanent_uuid: "81b6de3f7bcd4d1392983f510cfac45d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39949 } } peers { permanent_uuid: "c265276c53f34cf091117951d3f48e8d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34045 } }
I20260926 04:02:47.055054 23045 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8382c3742b944661b02a6f3bae8b3ffc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8382c3742b944661b02a6f3bae8b3ffc, State: Initialized, Role: FOLLOWER
I20260926 04:02:47.055135 23045 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8382c3742b944661b02a6f3bae8b3ffc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8382c3742b944661b02a6f3bae8b3ffc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43783 } } peers { permanent_uuid: "81b6de3f7bcd4d1392983f510cfac45d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39949 } } peers { permanent_uuid: "c265276c53f34cf091117951d3f48e8d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34045 } }
I20260926 04:02:47.055512 23019 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8382c3742b944661b02a6f3bae8b3ffc [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 04:02:47.055581 23019 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8382c3742b944661b02a6f3bae8b3ffc [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:02:47.056406 23019 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8382c3742b944661b02a6f3bae8b3ffc [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8382c3742b944661b02a6f3bae8b3ffc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43783 } } peers { permanent_uuid: "81b6de3f7bcd4d1392983f510cfac45d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39949 } } peers { permanent_uuid: "c265276c53f34cf091117951d3f48e8d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34045 } }
I20260926 04:02:47.056690 23019 leader_election.cc:288] T 00000000000000000000000000000000 P 8382c3742b944661b02a6f3bae8b3ffc [CANDIDATE]: Term 1 election: Requested vote from peers 81b6de3f7bcd4d1392983f510cfac45d (127.0.98.61:39949), c265276c53f34cf091117951d3f48e8d (127.0.98.60:34045)
I20260926 04:02:47.057035 23169 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8382c3742b944661b02a6f3bae8b3ffc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c265276c53f34cf091117951d3f48e8d"
I20260926 04:02:47.057161 23169 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c265276c53f34cf091117951d3f48e8d [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:02:47.057161 23094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8382c3742b944661b02a6f3bae8b3ffc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "81b6de3f7bcd4d1392983f510cfac45d"
I20260926 04:02:47.057284 23094 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81b6de3f7bcd4d1392983f510cfac45d [term 0 FOLLOWER]: Advancing to term 1
I20260926 04:02:47.058109 23169 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c265276c53f34cf091117951d3f48e8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8382c3742b944661b02a6f3bae8b3ffc in term 1.
I20260926 04:02:47.058494 22993 leader_election.cc:302] T 00000000000000000000000000000000 P 8382c3742b944661b02a6f3bae8b3ffc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8382c3742b944661b02a6f3bae8b3ffc, c265276c53f34cf091117951d3f48e8d; no voters:
I20260926 04:02:47.058517 23094 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 81b6de3f7bcd4d1392983f510cfac45d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8382c3742b944661b02a6f3bae8b3ffc in term 1.
I20260926 04:02:47.058810 23230 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8382c3742b944661b02a6f3bae8b3ffc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 04:02:47.059000 23230 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8382c3742b944661b02a6f3bae8b3ffc [term 1 LEADER]: Becoming Leader. State: Replica: 8382c3742b944661b02a6f3bae8b3ffc, State: Running, Role: LEADER
I20260926 04:02:47.059127 23230 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8382c3742b944661b02a6f3bae8b3ffc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8382c3742b944661b02a6f3bae8b3ffc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43783 } } peers { permanent_uuid: "81b6de3f7bcd4d1392983f510cfac45d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39949 } } peers { permanent_uuid: "c265276c53f34cf091117951d3f48e8d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34045 } }
I20260926 04:02:47.055465 23045 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8382c3742b944661b02a6f3bae8b3ffc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 04:02:47.059702 23230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8382c3742b944661b02a6f3bae8b3ffc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8382c3742b944661b02a6f3bae8b3ffc. Latest consensus state: current_term: 1 leader_uuid: "8382c3742b944661b02a6f3bae8b3ffc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8382c3742b944661b02a6f3bae8b3ffc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43783 } } peers { permanent_uuid: "81b6de3f7bcd4d1392983f510cfac45d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39949 } } peers { permanent_uuid: "c265276c53f34cf091117951d3f48e8d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34045 } } }
I20260926 04:02:47.059854 23230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8382c3742b944661b02a6f3bae8b3ffc [sys.catalog]: This master's current role is: LEADER
I20260926 04:02:47.060174 23239 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 04:02:47.060719 23239 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 04:02:47.055460 23229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8382c3742b944661b02a6f3bae8b3ffc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8382c3742b944661b02a6f3bae8b3ffc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43783 } } peers { permanent_uuid: "81b6de3f7bcd4d1392983f510cfac45d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39949 } } peers { permanent_uuid: "c265276c53f34cf091117951d3f48e8d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34045 } } }
I20260926 04:02:47.061128 23229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8382c3742b944661b02a6f3bae8b3ffc [sys.catalog]: This master's current role is: FOLLOWER
I20260926 04:02:47.065548 23169 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c265276c53f34cf091117951d3f48e8d [term 1 FOLLOWER]: Refusing update from remote peer 8382c3742b944661b02a6f3bae8b3ffc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:02:47.065934 23230 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8382c3742b944661b02a6f3bae8b3ffc [LEADER]: Connected to new peer: Peer: permanent_uuid: "c265276c53f34cf091117951d3f48e8d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:02:47.066300 23094 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 81b6de3f7bcd4d1392983f510cfac45d [term 1 FOLLOWER]: Refusing update from remote peer 8382c3742b944661b02a6f3bae8b3ffc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 04:02:47.066735 23230 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8382c3742b944661b02a6f3bae8b3ffc [LEADER]: Connected to new peer: Peer: permanent_uuid: "81b6de3f7bcd4d1392983f510cfac45d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 04:02:47.067939 23215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81b6de3f7bcd4d1392983f510cfac45d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8382c3742b944661b02a6f3bae8b3ffc. Latest consensus state: current_term: 1 leader_uuid: "8382c3742b944661b02a6f3bae8b3ffc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8382c3742b944661b02a6f3bae8b3ffc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43783 } } peers { permanent_uuid: "81b6de3f7bcd4d1392983f510cfac45d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39949 } } peers { permanent_uuid: "c265276c53f34cf091117951d3f48e8d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34045 } } }
I20260926 04:02:47.068022 23215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81b6de3f7bcd4d1392983f510cfac45d [sys.catalog]: This master's current role is: FOLLOWER
I20260926 04:02:47.068982 23201 sys_catalog.cc:455] T 00000000000000000000000000000000 P c265276c53f34cf091117951d3f48e8d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8382c3742b944661b02a6f3bae8b3ffc. Latest consensus state: current_term: 1 leader_uuid: "8382c3742b944661b02a6f3bae8b3ffc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8382c3742b944661b02a6f3bae8b3ffc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43783 } } peers { permanent_uuid: "81b6de3f7bcd4d1392983f510cfac45d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39949 } } peers { permanent_uuid: "c265276c53f34cf091117951d3f48e8d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34045 } } }
I20260926 04:02:47.069067 23201 sys_catalog.cc:458] T 00000000000000000000000000000000 P c265276c53f34cf091117951d3f48e8d [sys.catalog]: This master's current role is: FOLLOWER
I20260926 04:02:47.070886 23124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:47.061144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50168 (local address 127.0.98.61:39949)
0926 04:02:47.061191 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:47.061193 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:47.062233 (+ 1040us) server_negotiation.cc:407] Connection header received
0926 04:02:47.062326 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:47.062327 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:47.062337 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:47.062369 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:47.063228 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:47.064007 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:47.065123 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:47.065414 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:47.065454 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:47.067074 (+ 1620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:47.067076 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:47.067076 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:02:47.067083 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:47.068815 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:47.070043 (+ 1228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:47.070044 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:47.070045 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:47.070104 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:47.070303 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:47.070304 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:47.070304 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:47.070428 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:47.070510 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:47.070714 (+ 204us) server_negotiation.cc:299] Negotiation successful
0926 04:02:47.070794 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 04:02:47.077769 23230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8382c3742b944661b02a6f3bae8b3ffc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8382c3742b944661b02a6f3bae8b3ffc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8382c3742b944661b02a6f3bae8b3ffc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43783 } } peers { permanent_uuid: "81b6de3f7bcd4d1392983f510cfac45d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39949 } } peers { permanent_uuid: "c265276c53f34cf091117951d3f48e8d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34045 } } }
I20260926 04:02:47.077888 23230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8382c3742b944661b02a6f3bae8b3ffc [sys.catalog]: This master's current role is: LEADER
I20260926 04:02:47.078038 23254 mvcc.cc:205] Tried to move back new op lower bound from 7333459423498326016 to 7333459423475486720. Current Snapshot: MvccSnapshot[applied={T|T < 7333459423498326016}]
I20260926 04:02:47.078097 23215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81b6de3f7bcd4d1392983f510cfac45d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8382c3742b944661b02a6f3bae8b3ffc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8382c3742b944661b02a6f3bae8b3ffc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43783 } } peers { permanent_uuid: "81b6de3f7bcd4d1392983f510cfac45d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39949 } } peers { permanent_uuid: "c265276c53f34cf091117951d3f48e8d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34045 } } }
I20260926 04:02:47.078167 23215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81b6de3f7bcd4d1392983f510cfac45d [sys.catalog]: This master's current role is: FOLLOWER
I20260926 04:02:47.078433 23253 mvcc.cc:205] Tried to move back new op lower bound from 7333459423498326016 to 7333459423475486720. Current Snapshot: MvccSnapshot[applied={T|T < 7333459423498326016}]
I20260926 04:02:47.079124 23230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8382c3742b944661b02a6f3bae8b3ffc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8382c3742b944661b02a6f3bae8b3ffc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8382c3742b944661b02a6f3bae8b3ffc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43783 } } peers { permanent_uuid: "81b6de3f7bcd4d1392983f510cfac45d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39949 } } peers { permanent_uuid: "c265276c53f34cf091117951d3f48e8d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34045 } } }
I20260926 04:02:47.079209 23230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8382c3742b944661b02a6f3bae8b3ffc [sys.catalog]: This master's current role is: LEADER
I20260926 04:02:47.079717 23201 sys_catalog.cc:455] T 00000000000000000000000000000000 P c265276c53f34cf091117951d3f48e8d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8382c3742b944661b02a6f3bae8b3ffc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8382c3742b944661b02a6f3bae8b3ffc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43783 } } peers { permanent_uuid: "81b6de3f7bcd4d1392983f510cfac45d" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39949 } } peers { permanent_uuid: "c265276c53f34cf091117951d3f48e8d" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 34045 } } }
I20260926 04:02:47.079806 23201 sys_catalog.cc:458] T 00000000000000000000000000000000 P c265276c53f34cf091117951d3f48e8d [sys.catalog]: This master's current role is: FOLLOWER
I20260926 04:02:47.080070 23256 mvcc.cc:205] Tried to move back new op lower bound from 7333459423498326016 to 7333459423475486720. Current Snapshot: MvccSnapshot[applied={T|T < 7333459423498326016}]
I20260926 04:02:47.085048 23199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:47.061955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59940 (local address 127.0.98.60:34045)
0926 04:02:47.062006 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:47.062008 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:47.071256 (+ 9248us) server_negotiation.cc:407] Connection header received
0926 04:02:47.071518 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:47.071519 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:47.071530 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:47.071585 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:47.072368 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:47.073135 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:47.075835 (+ 2700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:47.076142 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:47.076184 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:47.079567 (+ 3383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:47.079569 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:47.079570 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:02:47.079577 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:47.081215 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:47.083658 (+ 2443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:47.083660 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:47.083660 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:47.083714 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:47.083900 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:47.083901 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:47.083902 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:47.084034 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:47.084103 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:47.084861 (+ 758us) server_negotiation.cc:299] Negotiation successful
0926 04:02:47.084941 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 04:02:47.086086 23048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:47.061650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37346 (local address 127.0.98.62:43783)
0926 04:02:47.061688 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:47.061690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:47.069282 (+ 7592us) server_negotiation.cc:407] Connection header received
0926 04:02:47.069377 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:47.069378 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:47.069388 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:47.069418 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:47.074160 (+ 4742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:47.074870 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:47.077794 (+ 2924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:47.078101 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:47.078142 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:47.081602 (+ 3460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 04:02:47.081604 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 04:02:47.081604 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 04:02:47.081611 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 04:02:47.083284 (+ 1673us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:47.084292 (+ 1008us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:47.084293 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:47.084294 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:47.084347 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 04:02:47.084533 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 04:02:47.084534 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 04:02:47.084534 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 04:02:47.084698 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 04:02:47.084786 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:47.085901 (+ 1115us) server_negotiation.cc:299] Negotiation successful
0926 04:02:47.085994 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 04:02:47.087523 23239 catalog_manager.cc:1347] Generated new cluster ID: 86c0866d54cd4920bed6c6a0efbcf4b8
I20260926 04:02:47.087580 23239 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 04:02:47.108724 23239 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 04:02:47.109443 23239 catalog_manager.cc:1504] Loading token signing keys...
I20260926 04:02:47.117767 23239 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8382c3742b944661b02a6f3bae8b3ffc: Generated new TSK 0
I20260926 04:02:47.117942 23239 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 04:02:47.143075 23124 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:02:47.141627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50170 (local address 127.0.98.61:39949)
0926 04:02:47.141675 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:47.141677 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:47.142361 (+ 684us) server_negotiation.cc:407] Connection header received
0926 04:02:47.142472 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:47.142473 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:47.142484 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:47.142517 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:47.142987 (+ 470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50170: BlockingRecv error: recv got EOF from 127.0.0.1:50170 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260926 04:02:47.143833 23048 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:02:47.142067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37360 (local address 127.0.98.62:43783)
0926 04:02:47.142108 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:47.142110 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:47.143315 (+ 1205us) server_negotiation.cc:407] Connection header received
0926 04:02:47.143414 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:47.143415 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:47.143427 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:47.143457 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:47.143768 (+ 311us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37360: BlockingRecv error: recv got EOF from 127.0.0.1:37360 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260926 04:02:47.144857 23199 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:02:47.144165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59942 (local address 127.0.98.60:34045)
0926 04:02:47.144203 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:47.144205 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:47.144324 (+ 119us) server_negotiation.cc:407] Connection header received
0926 04:02:47.144419 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:47.144420 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:47.144431 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:47.144460 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:47.144798 (+ 338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59942: BlockingRecv error: recv got EOF from 127.0.0.1:59942 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260926 04:02:47.147244 23124 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:02:47.146426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50184 (local address 127.0.98.61:39949)
0926 04:02:47.146464 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:47.146465 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:47.146671 (+ 206us) server_negotiation.cc:407] Connection header received
0926 04:02:47.146775 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:47.146776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:47.146794 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:47.146824 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:47.147199 (+ 375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50184: BlockingRecv error: recv got EOF from 127.0.0.1:50184 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260926 04:02:47.150260 23124 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:02:47.149475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50186 (local address 127.0.98.61:39949)
0926 04:02:47.149508 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:47.149509 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:47.149759 (+ 250us) server_negotiation.cc:407] Connection header received
0926 04:02:47.149856 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:47.149857 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:47.149867 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:47.149896 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:47.150214 (+ 318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50186: BlockingRecv error: recv got EOF from 127.0.0.1:50186 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260926 04:02:47.154058 23124 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:02:47.153301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50196 (local address 127.0.98.61:39949)
0926 04:02:47.153335 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:47.153337 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:47.153571 (+ 234us) server_negotiation.cc:407] Connection header received
0926 04:02:47.153667 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:47.153668 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:47.153678 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:47.153706 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:47.154003 (+ 297us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50196: BlockingRecv error: recv got EOF from 127.0.0.1:50196 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260926 04:02:47.159945 23124 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:02:47.159147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50206 (local address 127.0.98.61:39949)
0926 04:02:47.159181 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:47.159183 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:47.159405 (+ 222us) server_negotiation.cc:407] Connection header received
0926 04:02:47.159504 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:47.159505 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:47.159515 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:47.159545 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:47.159902 (+ 357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50206: BlockingRecv error: recv got EOF from 127.0.0.1:50206 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260926 04:02:47.171552 23124 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:02:47.169423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50214 (local address 127.0.98.61:39949)
0926 04:02:47.169471 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:47.169473 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:47.170967 (+ 1494us) server_negotiation.cc:407] Connection header received
0926 04:02:47.171087 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:47.171088 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:47.171101 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:47.171132 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:47.171491 (+ 359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50214: BlockingRecv error: recv got EOF from 127.0.0.1:50214 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260926 04:02:47.189885 23124 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:02:47.188993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50216 (local address 127.0.98.61:39949)
0926 04:02:47.189038 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:47.189041 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:47.189318 (+ 277us) server_negotiation.cc:407] Connection header received
0926 04:02:47.189420 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:47.189421 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:47.189433 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:47.189462 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:47.189825 (+ 363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50216: BlockingRecv error: recv got EOF from 127.0.0.1:50216 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 04:02:47.224073 23124 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:02:47.223321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50226 (local address 127.0.98.61:39949)
0926 04:02:47.223380 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:47.223383 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:47.223472 (+ 89us) server_negotiation.cc:407] Connection header received
0926 04:02:47.223525 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:47.223526 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:47.223540 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:47.223627 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:47.223990 (+ 363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50226: BlockingRecv error: recv got EOF from 127.0.0.1:50226 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260926 04:02:47.289793 23124 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:02:47.289124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50230 (local address 127.0.98.61:39949)
0926 04:02:47.289213 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:47.289215 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:47.289296 (+ 81us) server_negotiation.cc:407] Connection header received
0926 04:02:47.289343 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:47.289344 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:47.289355 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:47.289418 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:47.289746 (+ 328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50230: BlockingRecv error: recv got EOF from 127.0.0.1:50230 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260926 04:02:47.419795 23124 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:02:47.419014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50246 (local address 127.0.98.61:39949)
0926 04:02:47.419110 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:47.419113 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:47.419215 (+ 102us) server_negotiation.cc:407] Connection header received
0926 04:02:47.419314 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:47.419315 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:47.419327 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:47.419399 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:47.419731 (+ 332us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50246: BlockingRecv error: recv got EOF from 127.0.0.1:50246 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260926 04:02:47.677764 23124 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 04:02:47.677008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50262 (local address 127.0.98.61:39949)
0926 04:02:47.677100 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:47.677102 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:47.677115 (+ 13us) server_negotiation.cc:407] Connection header received
0926 04:02:47.677193 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:47.677194 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:47.677205 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:47.677265 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 04:02:47.677692 (+ 427us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50262: BlockingRecv error: recv got EOF from 127.0.0.1:50262 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 04:02:47.988257 23212 catalog_manager.cc:1259] Loaded cluster ID: 86c0866d54cd4920bed6c6a0efbcf4b8
I20260926 04:02:47.988332 23212 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c265276c53f34cf091117951d3f48e8d: loading cluster ID for follower catalog manager: success
I20260926 04:02:47.989113 23212 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c265276c53f34cf091117951d3f48e8d: acquiring CA information for follower catalog manager: success
I20260926 04:02:47.989447 23212 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c265276c53f34cf091117951d3f48e8d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 04:02:48.028906 23226 catalog_manager.cc:1259] Loaded cluster ID: 86c0866d54cd4920bed6c6a0efbcf4b8
I20260926 04:02:48.029002 23226 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 81b6de3f7bcd4d1392983f510cfac45d: loading cluster ID for follower catalog manager: success
I20260926 04:02:48.029803 23226 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 81b6de3f7bcd4d1392983f510cfac45d: acquiring CA information for follower catalog manager: success
I20260926 04:02:48.030112 23226 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 81b6de3f7bcd4d1392983f510cfac45d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 04:02:48.193718 23324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:48.190847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50278 (local address 127.0.98.61:39949)
0926 04:02:48.190971 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:48.190974 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:48.191069 (+ 95us) server_negotiation.cc:407] Connection header received
0926 04:02:48.191188 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:48.191189 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:48.191261 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:48.191349 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 04:02:48.191523 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:48.192028 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:48.192875 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:48.193085 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:48.193162 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:48.193335 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 04:02:48.193458 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 04:02:48.193542 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:48.193595 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 04:02:48.193618 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":57,"threads_started":1}
I20260926 04:02:48.197561 23330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:48.195344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59956 (local address 127.0.98.60:34045)
0926 04:02:48.195435 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:48.195437 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:48.195445 (+ 8us) server_negotiation.cc:407] Connection header received
0926 04:02:48.195503 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:48.195504 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:48.195535 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:48.195603 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 04:02:48.195689 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:48.196166 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:48.196890 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:48.197081 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:48.197133 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:48.197239 (+ 106us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 04:02:48.197324 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 04:02:48.197387 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:48.197430 (+ 43us) server_negotiation.cc:299] Negotiation successful
0926 04:02:48.197446 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72,"thread_start_us":46,"threads_started":1}
I20260926 04:02:48.201972 23330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:48.199230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59960 (local address 127.0.98.60:34045)
0926 04:02:48.199268 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:48.199270 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:48.199438 (+ 168us) server_negotiation.cc:407] Connection header received
0926 04:02:48.199495 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:48.199496 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:48.199505 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:48.199553 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 04:02:48.199734 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:48.200190 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:48.201216 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:48.201494 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:48.201567 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:48.201706 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 04:02:48.201733 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 04:02:48.201781 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:48.201880 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 04:02:48.201906 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 04:02:48.202122 23324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:48.199174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50282 (local address 127.0.98.61:39949)
0926 04:02:48.199214 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:48.199216 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:48.199419 (+ 203us) server_negotiation.cc:407] Connection header received
0926 04:02:48.199515 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:48.199516 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:48.199546 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:48.199576 (+ 30us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 04:02:48.199765 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:48.200462 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:48.201305 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:48.201600 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:48.201633 (+ 33us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:48.201869 (+ 236us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 04:02:48.201900 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 04:02:48.201929 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:48.202030 (+ 101us) server_negotiation.cc:299] Negotiation successful
0926 04:02:48.202049 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 04:02:48.202729 23336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 04:02:48.199164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37370 (local address 127.0.98.62:43783)
0926 04:02:48.199319 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 04:02:48.199320 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 04:02:48.199330 (+ 10us) server_negotiation.cc:407] Connection header received
0926 04:02:48.199414 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 04:02:48.199415 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 04:02:48.199468 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 04:02:48.199555 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 04:02:48.199643 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:48.200426 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:48.201393 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 04:02:48.201697 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 04:02:48.201761 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 04:02:48.201966 (+ 205us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 04:02:48.202502 (+ 536us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 04:02:48.202561 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 04:02:48.202649 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 04:02:48.202672 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":58,"threads_started":1}
I20260926 04:02:48.211647 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 22977
I20260926 04:02:48.216874 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 23052
I20260926 04:02:48.222008 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task7pf3XC/build/release/bin/kudu with pid 23127
2026-09-26T04:02:48Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2952 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5715 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (272754 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST